Release Notes 2019

Release notes for STRATUS Hosting 2019

Release Notes

Rss Feed

11-27-2019

Bug Fixes

STRAT-2072 ES 6 and 7 Crashloop;
STRAT-2084 In Prod store last overlay Loading… not visible;
STRAT-2089 Provision bug on dev cluster;
STRAT-2096 Sphinx service not accessible from cron container Indexes can not be updated;
STRAT-2104 When using SSL or Urls tabs - an error occurs periodically;
STRAT-2109 autoscaling not able to get cpu metrics;
STRAT-2155 Ticket 1237063 CPDoS attacks - is CloufFront used correctly configured to prevent;
STRAT-2160 reported instance type under info is wrong;
STRAT-2165 cron for LE autorenew missing;
STRAT-2198 databases missing SUPER privs

New Items

STRAT-1260 Pre-cache container images on worker nodes;
STRAT-1470 Add rate limit to wishlist path for Magento 1 Nginx template;
STRAT-1696 Add menu item mysql processlist;
STRAT-1945 Upgrade Redis to Redis 5;
STRAT-1957 Rework maintenance section for Magento 1;
STRAT-2129 Add nginx default allow for IP 3.230.118.30;
STRAT-2152 poc io control;
STRAT-2162 STRATUS Diagram;
STRAT-2195 update php versions for mm.com stack;
STRAT-2202 The button activity area goes beyond the inscription;
STRAT-2209 announce additional 24;
STRAT-2211 Magento 2.3.3 not compatible with SES;
STRAT-2212 Get list of customers without autoscaling on;
STRAT-2217 migrate sydney dev node;
STRAT-2226 Mysql Info shows Errors and no DB Information;
STRAT-2260 kube ephermeral storage location;
STRAT-2264 POC autoscaler;
STRAT-1991 Node to Node DR

11-19-2019

Bug Fixes

STRAT-2155 Fixed error responses quantity;
STRAT-2160 Fixed instance type on Info page;
STRAT-2167 Fixed mysql restart procedure

New Items

STRAT-2096 Changed process lookup for elasticsearch, memcached, sphinx, and varnish;
STRAT-1696 Changed MySQL user from store db user to root;
STRAT-2037 Blackfire service redesign;
STRAT-2155 Added custom error responses with ttl=0 to prevent CPDoS attacks;
STRAT-1696 Added ‘show full processlist’ procedure;
STRAT-2084 Added ‘no reload’ option for object jobs. Added store reload notification for DNS procedures;
STRAT-1957 Added Magento maintenance management functionality;
STRAT-2135 Added phpenv;
STRAT-1966 Changed IAM user for AWS API access;
STRAT-1696+STRAT-2202 restyled helper links under form elements;
STRAT-1696 Added MySql->Processlist tab;
STRAT-2084 refactoring Loading… overlay and objectAWSWampCall calls. Add ‘updating store’ messages near the headers of all pages;
STRAT-1957 Refactored calls to get/set maintenance mode;
STRAT-1906 Added info text to IP field on SSH->Whitelist;
STRAT-2086 Fixes REMOTE_ADDR variable being 127.0.0.1 when Varnish is enabled, Important for various modules that use that variable

11-08-2019

Bug Fixes

STRAT-1813 Php service says disabled;
STRAT-2044 Rabbitmq authentication issues;
STRAT-2077 Sphinx configuration path needs the config file name;

New Items

STRAT-1652 STRATUS helpdesk links;
STRAT-2076 Install poc to customer dev nodes;
STRAT-912 Create guide on editing and updating docker images;
STRAT-1504 Remove MageMojo S3 buckets from list if the customer uses the aws cli;
STRAT-1533 Add optipng to the fpm container;
STRAT-1542 Add re-label configuration to cluster prometheus;
STRAT-1562 Kube state metrics not working;
STRAT-1576 cwebp binary installation request;
STRAT-1643 Update ssh keys used to access workers example aws_kubis;
STRAT-1709 Migrations changing limits etc;
STRAT-1730 Timestamps in SSH bash history;
STRAT-1761 Jenkins Jira integration via Jenkinsfile Scripted Pipeline;
STRAT-1779 In Elasticsearch Info can not enable service ( enable/disable buttons are locked );
STRAT-1846 Change display_errors default for PHP frontend;
STRAT-1847 Remove heartbeat link from mhm ui;
STRAT-1849 Create deployment road-map and structure for MageMojo Infra/Code production and development;
STRAT-1964 Install gsfonts on php cron and ssh containers;
STRAT-1892 mhm-qa Jenkinsfile and add to new jenkins;
STRAT-1916 Php 7.3 support;
STRAT-2036 Remove WAF ACL from STRATUS Configurations;
STRAT-2043 RabbitMQ Info Page gives KWARGS pop-up if RabbitMQ is disabled;
STRAT-2046 CLI is not working on Virginia Dev sites;
STRAT-2073 New provision can overwrite Route53 zone file;
STRAT-2093 sphinx.conf file always default after restart;
STRAT-2128 Storage billing fix in AU;
STRAT-2144 The build version in the footer must be 2.11 not 2.10;
STRAT-2184 php-fpm rlimit_core;
STRAT-2194 Update php versions for latest security fixes

09-13-2019

New Items

STRAT-161 RabbitMQ;
STRAT-1950 nginx security update;
STRAT-1181 various fixes for sphinx issues;
STRAT-1352 elasticsearch 6 and 7 support;
STRAT-1510 crons to run as low priority;
STRAT-1911 fix for domain file verification;
STRAT-1914 Access > Block Bots> When I click button Cancel data disappears;
STRAT-2035 php upgrade for security vulnerability

08-16-2019

New Items

STRAT-1948 implement IPVS;
STRAT-1483 Kubenetes Upgrade;
STRAT-1485 implement Network ACLs;
STRAT-1490 Update Migration Script;
STRAT-1491 ELB removal;
STRAT-1762 retire london region;
STRAT-1764 implement kiam;
STRAT-1765 update ubuntu distro to latest version;
STRAT-1766 kubenetes in place upgrades;
STRAT-1767 script cluster install in terraform;
STRAT-1773 remove objfs from infrastructure;
STRAT-1775 dev nodes to move to i3en instances;
STRAT-1827 pipeline for setting up monitoring of burst credits for new nodes st1 drives;
STRAT-1860 sydney instance types update;
STRAT-1861 vpn access to frankfurt from us-east;
STRAT-1880 proper placement of coredns;
STRAT-1949 ssh hostname update

07-03-2019

Bug Fixes

STRAT-1791 Bug - checkmarks should not be green by default;
STRAT-1793 Bug - why does total snapshot show 17mb?;
STRAT-1794 Bug - when I click on a row it should select that row (and unselect any others);
STRAT-1795 Bug - the time needs to show the timezone;
STRAT-1796 Bug - remove filter for size;
STRAT-1797 Bug - total needs to have MB by the size;
STRAT-1802 Bug - alert when files/database snaps will be removed;
STRAT-1806 Bug - remove the word “snapshot” from all the header columns;
STRAT-1807 Bug - total at bottom should round to 2 decimal places

New Items

STRAT-100 block 0.0.0.0/0 in whitelist;
STRAT-1363 Restore Snapshot;
STRAT-1583 nginx worker_connections exceed;
STRAT-1640 Put MOTD updates into pipeline so changes to master MOTD are pushed out;
STRAT-1661 open files limit for memcached;
STRAT-1662 Jenkins Docker Container Build;
STRAT-1886 need php module libsodium installed;
STRAT-1682 Make sure that all body fonts are at least 14px;
STRAT-1699 Add hotjar tracking js to ui;
STRAT-1731 Panel logs for customer/user panel actions;
STRAT-1744 backup monitoring;
STRAT-1782 A small visual bug in dns;
STRAT-1784 In Add SSH User there is a button Delete;
STRAT-1785 upgrade php 7.2 to latest version 7.2.19;
STRAT-1810 Add pm static | dynamic dropdown to admin -> php-fpm;
STRAT-1811 php-fpm default settings are bad;
STRAT-1814 docker build with dependencies;
STRAT-1829 Snapshots validation for unique name does not work for everyone Scope;
STRAT-1832 General snapshots testing;
STRAT-1836 Add docker image prefix as a parameter to cluster pillar config;
STRAT-1837 Update UI Redis Tab to show all 3 Redis instance info like config port and flush cache button;
STRAT-1838 magedetect is not properly handling symlinks;
STRAT-1844 Ticket #1095661 Server Time;
STRAT-1851 KB Limits don’t match Google Sheets;
STRAT-1888 redis containers ui show enabled for containers by default, even if they are not;
STRAT-1891 aditional redis containers do not always resolve;
STRAT-1893 Build php 7.1, 7.2 containers from master prior to deployment;
STRAT-1895 TAG_PREFIX should be empty for DockerBuild on master branch

06-26-2019

Bug Fixes

STRAT-1678 403 on logs download in AU;
STRAT-1732 502 on logs download in NV;
STRAT-1738 autoscaling -> redeploy code button should do reinit instead of redeploy;
STRAT-1736 storage api incorrect;
STRAT-1709 Migrations changing limits and other strange behavior;
STRAT-1743 AWS rate limit error on Route53 updates during LetsEncrypt generation

New Items

STRAT-1421 Enhance the Info page with quick read-only info about the customer;
STRAT-1668 additional info in Admin pods menu;
STRAT-1673 Dns > When you search some object button Add MX Record not working;
STRAT-1675 Change default autoscaling configuration for m1;
STRAT-1687 Changing document root w/ autoscaling on does not redeploy code;
STRAT-1695 Unify the look of pages where header duplicates the only form label;
STRAT-1705 Improve speed of domain verification;
STRAT-1711 Add CloudFront Behavior for .ico files

06-03-2019

Bug Fixes

STRAT-1701 Google invalidation issue;
STRAT-1733 suspension process to open ticket instead of autosuspend

New Items

STRAT-1710 kernel update 4.18.0-1016-aws;
STRAT-1748 make frankfurt great again;
STRAT-1755 Add admin section for php-fpm values

05-20-2019

New Items

STRAT-1655 enhance noc for stratus malware data;
STRAT-1726 expose magedetect info for stratus in noc

05-13-2019

Bug Fixes

STRAT-1685 fix mwscan

New Items

STRAT-161 Ticket #705555 RabbitMQ;
STRAT-912 Create guide on editing and updating docker images;
STRAT-1260 Pre-cache container images on worker nodes;
STRAT-1261 As an employee I want us to follow Kubernetes best practices for container image pulls;
STRAT-1352 add support for ES 6.0;
STRAT-1639 Collect modules name and version;
STRAT-1644 Is autoscaling enabled;
STRAT-1645 Are any countries blocked;
STRAT-1665 change us-east shared storage nodes to use r5d.12xlarge instance type;
STRAT-1671 automatically disable magemojo admin users for stores;
STRAT-1763 remove objfs for store configs

05-06-2019

New Items

STRAT-1570 Updated Jenkinsfile;
STRAT-1564 Updated provision procedure. Updated CloudFront config render to use only those domains that are covered by the used SSL certificate. Domain Is Covered by SSL Certificate logic BUGFIX fixed typo k8s.py;
STRAT-1665 change us-east shared storage nodes to use r5d.12xlarge instance type;
STRAT-1666 launch new shared storage instance on r5d.12xlarge instance type;
STRAT-1667 Set redis.conf maxmemory-policy allkeys-lru

04-29-2019

Bug Fixes

STRAT-1251 Can’t see container status reliably

New Items

STRAT-1286 Set TLS default to 1.2;
STRAT-1301 Disallow magemojo as SSH username;
STRAT-1403 Crons and restricted mode;
STRAT-1650 mysql cli copy connection details (like ssh user)

04-22-2019

Bug Fixes

STRAT-1563 doublesave bug strikes again

New Items

STRAT-1522 update autoscaling cpu target;
STRAT-1330 Dev stores are running in production mode;
STRAT-1361 Update imagemagic version on stratus;
STRAT-1454 [Ticket #1040232] S3 Bucket Backup;
STRAT-1558 mysql limits;
STRAT-1590 Ticket #1053289 Dev Site will not start;
STRAT-1614 provision new shared worker

04-15-2019

Bug Fixes

STRAT-1579 fix pod_exec_cmd() status processing;
STRAT-1573 Fixed Admin Only->Varnish values dropping to default after updating Varnish->VCL tab;
STRAT-1574 Fixed minor routing problems;
STRAT-1566 Removed state call duplication, fix memcached cache flush command;
STRAT-225 HOTFIX # ssh/nginx logging create log dir for existing pods rework of ssh+nginx logs for ssh logging;
STRAT-1397 STRATUS Logs can not select file to download;
STRAT-1511 Fixed changes order in Route53 update batches;
STRAT-1468 Sydney Exim log permissions missing a+r;
STRAT-1553 Fixed error handling on Magento->Info and MySQL->Info tabs. Changed Jenkinsfile;
STRAT-1534 Fixed DNS Zone creation Cancel button;
STRAT-1535 Fixed SSL logic for expired LE certificates

New Items

STRAT-629 move mysql logs to /log/mysql;
STRAT-943 Setting store name equal to provision domain;
STRAT-1003 fix errors processing on pod exec;
STRAT-1006 making node load checking tools;
STRAT-1144 add bots filtering for development mode;
STRAT-1205 GeoIP PHP module for all PHP versions;
STRAT-1318 Ticket 977143 Install Image Packages - jpegoptim;
STRAT-1357[Ticket 994454 STRATUS Migration Request: PDO DBLIB;
STRAT-1373 STRAT-1464 alter php default values;
STRAT-1443 Added Clear URL Cache page in Varnish section;
STRAT-1455 poc enhancements for killing wild crons and mysql processes;
STRAT-1473 Ticket #1044754 Node.js version upgrade request;
STRAT-1474 Update Image Magick Resource Limits;
STRAT-1475 System Log settings | Log rotation not in effect;
STRAT-1476 Temporary hack for flexvolumes for new single pool nodes until we finish plans development;
STRAT-1477 Migration script to support dedicated nodes;
STRAT-1478 Growth Plan Metal Nodes;
STRAT-1479 Scale and Enterprise plans;
STRAT-1499 Ticket #1046415 Install openssl-dev;
STRAT-1500 Ticket #1046424 Install pdftk;
STRAT-1508 cloudfront download logs error;
STRAT-1514 Use defaults for blocked bots if the list is empty;
STRAT-1515 Hid Autoscaling for rbp plans;
STRAT-1518 Update n98 to support magento 2.3.1;
STRAT-1528 Multiple SRV records Fixed adding DNS on provision, Restructured SRV record;
STRAT-1541 Blocked LetsEncrypt option on SSL->Settings tab if there are no verified domains;
STRAT-1548 Changed STRATUS plan to STRATUS Starter;
STRAT-1567 improve poc logging;
STRAT-1568 STRAT-1214 Refactoring varnish service, Added Admin->Varnish page in order to manage varnish Cache Size and Configuration Added Varnish config and cache_size fields Updated SaltStack error handling;
STRAT-1581 Updated SaltStack error handling;
STRAT-1582 Banned wildcard from Add URL input on URLs page

04-08-2019

Bug Fixes

STRAT-1003 fix errors processing on pod exec;
STRAT-1511 Fixed changes order in Route53 update batches;
STRAT-1534 Fixed DNS Zone creation Cancel button;
STRAT-1535 Fixed SSL logic for expired LE certificates

New Items

STRAT-225 rework of ssh+nginx logs for ssh logging;
STRAT-629 move mysql logs to /log/mysql;
STRAT-1144 add bots filtering for development mode;
STRAT-1373 STRAT-1464 alter php default values;
STRAT-1443 Added Clear URL Cache page in Varnish section;
STRAT-1514 Use defaults for blocked bots if the list is empty;
STRAT-1515 Hid Autoscaling for rb” plans;
STRAT-1528 Multiple SRV records Fixed adding DNS on provision;
STRAT-1528 Restructured SRV record;
STRAT-1541 Blocked ‘LetsEncrypt’ option on SSL->Settings tab if there are no verified domains;
STRAT-1548 Changed STRATUS plan to STRATUS Starter

04-01-2019

Bug Fixes

STRAT-1246 As a customer I have to save anything Nginx related twice all the time;
STRAT-1496 can not renew SSL with weird error;
STRAT-1414 Provision preprovision fixes

New Items

STRAT-1095 Wanting to change www CNAME of stratus DNS;
STRAT-1230 document adding additional ports for provisioning;
STRAT-1317 STRATUS panel CLI throwing filedescriptor out of range;
STRAT-1443 Added Clear URL Cache page in Varnish section;
STRAT-1477 Migration script to support dedicated nodes;
STRAT-1494 Block SES;
STRAT-1511 Fixed changes order in Route53 update batches;
STRAT-1525 REST API endpoint to cleanup ACM;
STRAT-1535 Fixed SSL logic for expired LE certificates

03-25-2019

Bug Fixes

STRAT-1065 Disable zfs scrub on node creation;
STRAT-1085 Alert on hugepages not being disabled;
STRAT-1085 Disable hugepages;
STRAT-1088 STRAT-1224 STRAT-1257 Enable Swap, kubelet and kubeadm;
STRAT-1146 Change tmp locations to customer zfs shares;
STRAT-1221 l2arc cache files in unavailable state after node stop;
STRAT-1224 Swap and cache settings not persisting after node stop;
STRAT-1257 142f reboot 1/29/2019 Ref: New Kernel linux-crashdump zfs kvm ena swap smb;
STRAT-1257 Install kdump and record crashes in /var/crash/;
STRAT-1257 KVM double_fault bug https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1744199/comments/3;
STRAT-1257 Update ENA driver to 2.0.2K https://github.com/amzn/amzn-drivers/blob/master/kernel/linux/ena/RELEASENOTES.md;
STRAT-1257 Update Kernel to 4.15.0-1032-aws which will update ENA, ZFS, KVM drivers info below / New AMI builds, disable unattended upgrades related;
STRAT-1257 Update ZFS driver to 0.7.5-1ubuntu16.4 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1781364/comments/25;
STRAT-1266 Remove Samba confirmed to cause Kernel Panics ref: Marty Simmons Eric Hileman Aleks Loz;
STRAT-1270 Larger root device / 1Tb to curb kicking nodes on disc pressure;
STRAT-1270 ephemeral storage pod eviction;
STRAT-1272 Broken /stratus/cache and /var/lib/docker mounts in AU;
STRAT-1282 /tmp setting a symlink in each container to reroute /tmp to their /tmp;
STRAT-1307 Install nfs-kernel packages in launch template and in salt;
STRAT-1309 migration script with enhancements;
STRAT-1310 automated snapshots of st1 volumes;
STRAT-1313 Remove all 12x testing workers from dev cluster;
STRAT-1324 Mojo plans instance types and templates;
STRAT-1325 standardized launch template names;
STRAT-1334 Remove unattended upgrades;
STRAT-1334 Remove unattended-upgrades package from the template in All regions;
STRAT-1335 Varnish cache is going to emptyDir and thus to /var/lib/kubelet;
STRAT-1336 Orphaned dir in var/lib/kubelet;
STRAT-1345 i-0df35473dcb77c50e down would not restart by itself;
STRAT-1347 Dev-Flight for STRAT-1297 Migration to new nodes with limits and correct mounts;
STRAT-1367 Migration script not working 100% re: Flex volume;
STRAT-1367 STRAT-1309 STRAT-1297 Optimise Migration script, and optimise not lose files;
STRAT-708 Disable hugepages on new worker creation

New Items

Replaced Samba with NFS;
Disabled unattended upgrades causing different kernel versions with different ENA ZFS driver versions;
Increased the size to 6Tb of ST1 type EBS devices which ran out of burst credits when backups ran causing ZFS lockups and subsequent crashes;
Ensured that SWAP was enabled on reboots and new servers;
Installed Kdump which records crashes in /var/crash;
Set nvme_core.timeout value from the 30s to the recommended highest setting;
Ensured termination protection for instances and EBS devices;
New KB released, old MHM content removed, stratus guides updated for UI v2.10, release notes cleaned.;
Enabled additional real-time logs in UI;
Fixed an NGINX double save bug in UI;
Fixed a bug in kube’s coredns that is used for internal DNS resolution;
Added new sections in STRATUS UI CloudFront Sphinx Elasticsearch Varnish Redis Memcache Nginx MySQL PHP Logs Cron Magento;
Added mssql ssh2 for PHP and sass gem for Ruby;
Fixed Autoscaling and Symlinks issue;
Fixed a bug in Autoscaling exclude directories

Table of Contents

Release Notes 2019

Release Notes

11-27-2019

Bug Fixes

New Items

11-19-2019

Bug Fixes

New Items

11-08-2019

Bug Fixes

New Items

09-13-2019

New Items

08-16-2019

New Items

07-03-2019

Bug Fixes

New Items

06-26-2019

Bug Fixes

New Items

06-03-2019

Bug Fixes

New Items

05-20-2019

New Items

05-13-2019

Bug Fixes

New Items

05-06-2019

New Items

04-29-2019

Bug Fixes

New Items

04-22-2019

Bug Fixes

New Items

04-15-2019

Bug Fixes

New Items

04-08-2019

Bug Fixes

New Items

04-01-2019

Bug Fixes

New Items

03-25-2019

Bug Fixes

New Items

Feedback