Version 0.6.1
If you are running a version prior to 0.5.0 please upgrade.
Dreamhost VPS Manager (short: PsManager) is an open source tool that dynamically manages the size of the memory of your Dreamhost virtual private server.
Managing the size of your memory dynamically, has a certain number of advantages. The main advantage is that you can continue to serve your visitors during peak traffic periods by temporarily increasing available resources without having to pay the monthly fee for these resources, instead you are only charged by Dreamhost for the time the resources were made available to you. Conversely during low traffic periods your resources will be lowered saving you from having to pay for resources that you do not really need all the time. Basically this puts a 4000 MB RAM webserver at your disposal at a fraction of the price.
It also saves you from having to guess how much memory you need and from having to set your server's memory size manually. Dreamhost VPS Manager will do these things for you based on actual current usage.
This program can also notify you, via e-mail or your microblog, of changes to your VPS and reboots of your VPS.
Note: Not a Dreamhost customer? Save $97 by signing up to Dreamhost with our Dreamhost Promo Codes.
You will need an API key for the Dreamhost web panel.
You can get one here:
Web Panel API.
This API key needs to be allowed to access dreamhost_ps-set_size
and dreamhost_ps-reboot.
It is important for the proper functioning of PsManager that your API key
can access both
these items.
Note: It is also best practice to have a seperate API key for each application. Thus even if you already have a API key, do create a new one that will by used only for PsManager.
If you do not have a Shell user on your Dreamhost VPS, create one here:
Manage Users - Add a new user.
Type of User Account: Shell
Shell Type: /bin/bash
Server: <select appropriate Private Server>
Proceed to download the PsManager script and the configuration file:
psmanager.pl, psrebooter.pl and psmanager.conf.
All three of these files need to be placed in the same directory on your VPS, for example in a "bin" directory of any user on the private server you want to manage. Avoid placing this script in your cgi-bin directory or under your site's document root, it should not be accessible via your website.
The easiest way do the above is to do it directly from the command line of your VPS. Open an SSH connection to a user on your private server at type the following commands:
mkdir ~/bin 2>/dev/null; cd ~/bin
wget -N http://www.7is7.com/software/dreamhost/download/psmanager.tar.gz
tar -zxvf psmanager.tar.gz
If you did not follow the procedure above, check that the permissions are
in order: psmanager.pl and psrebooter.pl must be executable while
psmanager.conf should only be readable by the owner (or type:
chmod 700 psmanager.pl psrebooter.pl; chmod 600 psmanager.conf).
Note: Is it important that your configuration file is not readable by others as it contains your Dreamhost API key and may contain your Twitter or Identi.ca passwords.
When the above is done, edit the configuration (psmanager.conf)
file filling in your Dreamhost API key.
$key = 'ABCDEFGHIKLM';
Optionally you can have PsManager send notifications of memory size changes and reboots to you via E-mail and/or status updates on Twitter or Identi.ca. See "Tweeting and/or Mailing Memory Changes" below for more information about these options.
Microblog settings:
$microblog = 'Identica'; # 'Identica' or 'SuperTweet' (Twitter) $uname = 'my_server'; $pass = 'secret';
E-mail setting:
$email_rcpt = 'you@yourdomain.com';
Finally to get everything running, you'll need to set up your crontab to run psmanager.pl every minute, add the following line to your crontab (this example assumes the program is located in your "bin" directory):
* * * * * $HOME/bin/psmanager.pl >> /tmp/psmanager_cron.log 2>&1
We recommend to edit your crontab yourself directly on your server, i.e. not to edit it via the DH Control Panel. This is because the Dreamhost control panel adds wrapper code that can disrupt the proper functioning of PsManager.
You can edit the crontab directly with crontab -e, while
crontab -l will list the contents of your crontab.
(Read more about
crontabs
on Dreamhost on their wiki page.)
After collecting a minimum amount of data, PsManager will start to make decisions about what memory size your system needs.
When upgrading from an older version, one generally only needs to
replace psmanager.pl, as the configuration file can usually
stay the same. You can basically follow the procedure above without replacing
your original psmanager.conf file
The instruction below will only extract the perl program from the tar-file after downloading it:
cd ~/bin
wget -N http://www.7is7.com/software/dreamhost/download/psmanager.tar.gz
tar -zxvf psmanager.tar.gz psmanager.pl
By default Dreamhost servers use Californian time, as this is where they are located. However you may want to see PsManager report changes in an other timezone, like your own. Simply add a line like the one below to your configuration file:
$ENV{'TZ'} = 'Europe/Tallinn';
To find your timezone, you can either browse through the subdirectories
of /usr/share/timezone/ or use the interactive command line
tool tzselect.
By default Dreamhost Ps Manager will set a memory level between 300 MB and 4000 MB, which are respectively the minimum and maximum allowed by Dreamhost. You can set a higher minimum or lower maximum in the configuration file (the lines are commented out by default).
A sudden increase of popularity of your site (either legitimate or the result of foul play) can cause an increase in allocated memory and higher hosting costs. To avoid the risk of allocating memory beyond what you are prepared to pay for, it is recommended to change the maximum memory setting to a level that you feel comfortable with. An example setting could be:
$mem_max = 2000;
Conversly you can also increase the minimum level of memory. For instance to increase the minimum from 300 MB to 400 MB add this line to the configuration file:
$mem_min = 400;
If the memory changes made are often in a particular range it can help to raise the minimum a bit so as to avoid making downward changes that are very temporary and will not save you much money anyway. Given the limit of 30 changes per day per VPS, you will also be keeping them for more important things.
You can also create more (or less) space by applying a memory factor other than 1 (which is the default). For example if you want to add 50% to the calculated memory, set memory_factor to 1.5.
$memory_factor = 1.5;
In February and March 2010 Dreamhost significantly changed its memory configuration. A major change is that caching is now reported separately instead of being included in the total used memory. Cache memory is mainly used to speed up disk access (often accessed files are kept in memory). However cached memory is always available for processes when needed, and so it is in reality freely available memory.
If you want to reserve more memory for caching purposes you could consider to increase the allocated memory by a more significant factor. This is not needed for the proper functioning of your webserver but it can't hurt either and will speed up your server (and sites) by significantly speeding up disk access (both write and read).
To double the allocated memory:
$memory_factor = 2;
Version 0.4.0 was released when Dreamhost changed the memory setting yet again in March 2010. Since that time no extra memory got allocated, instead you had to set the total you wanted. The extra buffer we were given before for free disappeared, but the price of memory was halved at the same time. To compensate for the loss of this buffer, we now maintain a buffer of 50% by default to preserve burstability of server. However this can be changed; lower it if you want to live dangerously and risk running out of memory or raise it if you want more peace of mind, though we think 50% (the default) is about right.
$buffer_perc = 50; # Memory buffer in percent
Note: $buffer_perc and $memory_factor
are applied sequentially as follows:
first the actual memory used by processes is increased with an amount
sufficient to maintain a buffer of the specified percentage. That result
in turn is multiplied by $memory_factor to give us the amount of memory
we want to reserve.
At this time, dreamhost is unable to resize a VPS to a level below its actual usage plus cache. While cache is free memory that is available at all times for processes, for some reason when downsizing a VPS this can not be freed up.
The only option left to us is to force a reboot, which will clear the cache and allow the resize to be made. We have introduced a limit to control when to force a reboot. With $force_lim set to 20 for instance, psmanager is instructed to force a reboot if the amount of memory reserved would be more than 20% higher than what is optimal.
$force_lim = 20;
Although reboots are fast they still cause a bit of downtime, hence you can avoid reboots when you think they are not worth the amount of memory saved.
To prevent PsManager from rebooting your VPS, simply increase the limit to a very high percentage that will never be reached.
$force_lim = 1000;
Note: We are hoping for Dreamhost to provide us with an API call to clear the cache as that would remove the need for reboots. Consider voting for the relevant suggestion: API Call to clear the cache. A more detailed explanation of this problem can be found on the Dreamhost forum, in the thread entitled: "We had to reboot your VPS".
PsManager should be able to determine the server you want to manage automatically. If for some reason it does not, you can manually set the 'ps' variable in the configuration file to indicate the correct machine.
$ps = 'ps0000';
As of version 0.2.4 you can set complex Perl instructions in the configuration file, for example to increase the minimum required memory at certain times of the day. For example the code below will set the minimum memory to 400MB from 7pm (19:00) to midnight.
{
my ($sec,$min,$hour) = localtime();
if ($hour>=19 && $hour<=23) {
$mem_min = 400;
$memory_factor = 1;
}
}
You can have PsManager send you e-mails and/or status updates about memory changes and reboots.
A really cool feature (or at least it was cool when Twitter was still cool) is undoubtedly that you can follow changes to your Dreamhost VPS via Twitter or Identica. The idea is simple, you create a Twitter or Identi.ca account for your Dreamhost VPS and let PsManager update it. You then proceed to follow this account from your own Twitter or Identi.ca account. Each time the memory of your Dreamhost VPS changes you'll see a tweet about the change appear in your updates stream.
Note: As Twitter has closed down Basic Authorization for its API, direct Twitter updates no longer work. However Twitter can still be updated in two possible ways:
Unfortunately Twitter's new OAuth mechanism isn't very userfriendly (going via Identica is or SuperTweet a lot easier for anyone to set up), would take a significant effort to implement, and would add a lot of bloat to our software. We are hoping Twitter will offer a more userfriendly solution in the not too distant future, something as easy as SuperTweet.
You can check how PsManager tweets the changes it makes to my Dreamhost VPS:
It is possible to define your own template for the status update e-mails and tweets, by adding a line to the configuration file (the {tz} tag was added in version 0.5.0). Template example for memory size updates:
$tweet_template = "{ps}: {change} resized to {newsize}MB from {oldsize}MB at {time} [{tz}]";
Template example for detected reboots:
$reboot_template = "{ps}: rebooted at {time} [{tz}]";
Where:
{ps} = the name of your VPS
{newsize} = the new memory allocation in MB
{oldsize} = the old memory allocation in MB
{time} = the time at which this change occurred
{tz} = the timezone offset
{change} = A visual representation of the change with - and + signs, with
the number of signs rounded up to the next multiple of 100 MB.
for instance: +++ = increase of between 200 and 300 MB
Since version 0.6.1, it is possible to run PsManager in monitor only mode.
It will log actual memory usage, changes and reboots but not request any
changes itself. Just set $monitor_only to 1 in the configuration
file.
$monitor_only = 1;
If you are encountering problems, these are a few starting points to check:
Check that the cronjob is set properly and runs every minute:
crontab -l | grep psmanager
Check the output file for the PsManager cronjob. It should be empty if there are no problems. Errors will be sent here:
tail /tmp/psmanager_cron.log
Check the usage log maintained by PsManager:
tail ~/bin/psmanager.log
The columns are: <timestamp> <available memory> <used memory> <free memory> <cache memory> <optimal memory> <set memory> <event>. All memory in MB.
Optimal memory is the amount of memory PsManager
thinks you should have at this very moment.
The set memory column indicates that PsManager gave the
instruction to change the memory to the given level.
The first example shows how memory is changed from 400MB to 316MB, 316 was calculated to be the optimal size at that moment:
20101125075701 400 43 356 229 302 0 20101125075801 400 43 356 229 298 0 20101125075901 400 41 358 230 294 0 20101125080001 400 46 353 230 316 316 S 20101125080100 316 46 269 231 311 0 C 20101125080201 316 46 269 231 304 0 20101125080301 316 31 284 232 297 0
The second example shows how a sudden increase in usage (from 52MB to 165MB) and a drop in the cache from 117 MB to 55 MB:
20101125124901 300 54 245 116 276 0 20101125125001 300 52 247 117 273 0 20101125125101 300 165 134 55 286 0 20101125125200 300 66 233 209 294 0 20101125125301 300 54 245 236 293 0
List all events:
grep -e '[BCS]' ~/bin/psmanager.log
Events are: B = reBoot detected, C = memory Change detected, S = memory Set.
Verbose logging (implemented since version 0.5.0)
will keep a log in ~/bin/psmanager_verbose.log
with events and details of each run. This is useful to look for possible
problems or to keep a detailed log for future reference.
To turn on verbose logging add this line to the configuration file:
$verbose = 1;
To list the latest (40) entries in the log simply use tail:
tail -40 ~/bin/psmanager_verbose.logTo find recent error messages:
tail -1000 ~/bin/psmanager_verbose.log |grep " E:"
Each line in the verbose log has an ID character. They are as follows: A = Api, B = (re)Boot, D = Debug, E = Error, I = Info, L = Logfile, M = Mail, R = Report.
Note: you may want to clear out old verbose logs every once in a while.
If you are encountering problems, and need help, PsManager can generate a report, that you can copy and paste into an e-mail. This checks a number of settings and the things mentioned above, while not disclosing any sensitive information like account names, passwords and API keys. To generate this report run psmanager.pl from the command line with the "check" argument:
~/bin/psmanager.pl check
PsManager monitors memory usage over the last 60 minutes and calculates an optimal memory requirement from that. Technically it takes the average of the 8 highest memory usage peaks (discarding the two highest ones) in the last 60 minutes. Single peaks into the memory buffer are acceptable, but regular use of the memory buffer is not advisable as it increases the risk of running out of memory all together. This is why we make an effort to avoid depending on the memory buffer and keep reserved memory on the high end of actual usage.
Dreamhost limits us to making 30 changes per VPS per day to the memory size, as a result we have to be a bit careful with changes. To avoid running into this restriction PsManager will never make more than 10 changes in 8 hours. In order to avoid making too many up and down changes, downward changes will occur only if no changes have been made in the previous 60 minutes and the change exceeds 10% of current reserved memory. If more than 4 changes have already been made in an 8 hour period, we are even more careful and only downsize after 120 minutes. As a result in any 4 hour period, a minimum of 3 changes are always reserved for upward corrections.
As it is more important to maintain availability of your sites a change will be made upwards if 30 minutes have passed and it exceeds 10%. If the number of changes in the past 8 hours exceeds 4 we will instead wait for 1 hour to pass. Unless the required increase exceeds 20% after 10 minutes (indicating a possible surge in traffic to your sites).
PsManager will also never use the last two available resizes in a 8 hour period for a 10% change, but instead reserve it for an emergency upsizing of at least 20%, as availability of your site is of paramount importance.
Version 0.2.6 introduced the usage trend in the calculation, if memory usage is going up over an hour it will reserve a bit more than current usage would indicate, and conversely if it is going down it will reserve a bit less. Since 0.5.0. this variation can fluctuate between -15% and +15%.
Versions 0.3.0 and 0.4.0 deal with the continuing changes Dreamhost made to the memory setup of new virtual private servers. Thankfully those changes are now well behind, the new Dreamhost setup is much better for efficient management of memory allocation.
Version 0.4.3 removes the list_ps API calls, making
it possible to manage multiple VPS servers on the same account without
hitting the API rate limit. This version now also tweets actual memory
updates instead of requested updates. Actual updates can lag the request
and sometimes happen in several steps - that is now visible.
Sinve version 0.5.0 PsManager keeps the log for 8 hours, making it possible to change reserved memory faster if few changes are made over an 8 hour period. The log file now also keeps track of memory used for disk cacheing. Version 0.5.0 also added the timezone tag, verbose logging and the self check report. It also fixed a few minor bugs.
Version 0.5.1 introduced a new way to update Twitter (via SuperTweet), and version 0.5.2 to 0.5.4 fixed some minor bugs, changed some message layouts and optimized some code.
Version 0.5.5 fixes some minor bugs and includes a workaround for the problem of not being able to resize below the level of memory and cache. Hopefully that problem will be fixed by Dreamhost, at which time this workaround will be disabled. Version 0.5.6 adds the possibility to reboot a VPS if cache usage makes it impossible to resize to the optimal memory level.
The most visible new features in version 0.6.0 are the possibility to e-mail status updates and system reboot detection. Besides that it includes several improvements to help troubleshooting. In version 0.6.1 we introduced the monitoring only flag.
As an active user of the tool myself, I am constantly monitoring how it performs and occasionally make tweaks or add features. Suggestions based on your own experience are of course welcome. Some suggestions have led to new features being integrated (for example the templated tweets with the change visualization with + and - signs). Others have helped a lot with testing on their very active virtual servers allowing me to fine tune the program to better deal with those situations.
Guaranteed freedom of use and full transparency of code.
The PsManager program is released under an open source license (GNU-GPL v3), which means that you can use this script for free and make any modifications that you please. You are in no way dependent on the original author of this program for future maintenance and your right to use it never expires. The source code of the script is easily viewable (the script is written in perl), you can scrutinize it or have it scrutinized to convince yourself that it does what it says it does.
If you are using the Dreamhost VPS Manager script (PsManager), you may also be interested in using the Dreamhost VPS Rebooter script (PsRebooter) to automatically schedule reboots (which clears out any hanging processes).
There is a Dreamhost forum thread for this application where you can ask questions. If you are using or want to use PsManager feel free to contact me by E-mail about any problems or suggestions. If you don't want to do the installation of PsManager yourself, I can also do the whole installation for you on your VPS in exchange for a small fee.
Many features are the result of feedback from people who told me what they would like to be able to do, or what problems they encountered. Several bugs have been fixed thanks to user feedback. Feedback has been valuable to make PsManager what it is today, a special thanks to everybody who has sent feedback.
Note: If you want to be notified of important new releases of PsManager, or problems affecting PsManager, send me an E-mail and ask to be added to the mailing list. (It is still a small mailing list that I maintain manually.)
Bookmark/Share/Support :
|
|
|
|
|
|
|
|
|
|
|
7is7.com | Software | Otto | Travel Stories | Countdown Clock | Firefox | StatEye
New | About | Contact | Connect | Friends | Promotions | Copyright | Advertise
