問題描述
我在一個使用 ImageMagick 生成圖像的網(wǎng)站上工作.該站點每分鐘將收到數(shù)百個請求，使用 ImageMagick 執(zhí)行此操作會導(dǎo)致站點崩潰.
I am working on a site that uses ImageMagick to generate images. The site will get hundreds of request every minute and using ImageMagick to do this causes the site to crash.
所以我們實現(xiàn)了 Redis 和 Php-resque 來在單獨的服務(wù)器上在后臺生成 ImageMagick，這樣它就不會使我們的主要服務(wù)器崩潰.問題是完成圖像仍然需要很長時間.用戶可能希望等待最多 2-3 分鐘的圖像請求，因為服務(wù)器正忙于處理這些圖像.
So we implemented Redis and Php-resque to do the ImageMagick generating in the background on a seperate server so that it doesn't crash our main one. The problem is that it's still taking a very long time to get images done. A user might expect to wait up to 2-3 minutes for an image request because the server is so busy processing these images.
我不知道該給你什么信息，但我更多的是在尋求建議.我認為如果我們可以減少 ImageMagick 請求的初始處理時間，那么顯然這將有助于加快我們可以處理的圖像數(shù)量.
I am not sure what information to give you, but I'm more looking for advice. I think if we can cut down the initial process time for the ImageMagick request, then obviously this will help speed up the amount of images we can process.
以下是我們使用的 ImageMagick 腳本示例:
Below is a sample of the ImageMagick script that we use:
convert -size 600x400 xc:none ( ".$path."assets/images/bases/base_image_69509021433289153_8_0.png -fill rgb(255,15,127) -colorize 100% ) -composite ( ".$path."assets/images/bases/eye_image_60444011438514404_8_0.png -fill rgb(15,107,255) -colorize 100% ) -composite ( ".$path."assets/images/markings/marking_clan_8_marking_10_1433289499.png -fill rgb(255,79,79) -colorize 100% ) -composite ( ".$path."assets/images/bases/shading_image_893252771433289153_8_0.png -fill rgb(135,159,255) -colorize 100% ) -compose Multiply -composite ( ".$path."assets/images/highlight_image_629750231433289153_8_0.png -fill rgb(27,35,36) -colorize 100% ) -compose Overlay -composite ( ".$path."assets/images/lineart_image_433715161433289153_8_0.png -fill rgb(0,0,0) -colorize 100% ) -compose Over -composite ".$path."assets/generated/queue/tempt_preview_27992_userid_0_".$filename."_file.png
我的理論是，這需要很長時間的原因是對圖像進行著色的過程.有沒有辦法優(yōu)化這個過程?
My theory is that the reason this takes quite a long time is due to the process of colouring the images. Is there a way to optimise this process at all?
如果您有處理大量 imagemagick 進程的經(jīng)驗，或者可以看到一些非常簡單的方法來優(yōu)化我們的請求，我會非常感激.
Anyone who has some experience with handling heavy loads of imagemagick processes or can see some glaringly easy ways to optimise our requests, I'd be very greatful.
謝謝:)
推薦答案
你的命令實際上可以歸結(jié)為:
Your command actually boils down to this:
convert -size 600x400 xc:none
( 1.png -fill rgb(x,y,z) -colorize 100% ) -composite
( 2.png -fill rgb(x,y,z) -colorize 100% ) -composite
( 3.png -fill rgb(x,y,z) -colorize 100% ) -composite
( 4.png -fill rgb(x,y,z) -colorize 100% ) -composite
( 5.png -fill rgb(x,y,z) -colorize 100% ) -composite
( 6.png -fill rgb(x,y,z) -colorize 100% ) -composite
result.png
我的想法如下:
第 1 點:
在空白畫布上的第一個 -composite 似乎毫無意義 - 大概 1.png 是具有透明度的 600x400 PNG，因此您的第一行可以避免合成操作并保存16% 的處理時間改為:
The first -composite onto a blank canvas seems pointless - presumably 1.png is a 600x400 PNG with transparency, so your first line can avoid the compositing operation and save 16% of the processing time by changing to:
convert -background none 1.png -fill ... -colorize 100%
( 2.png ..
( 3.png ...
第 2 點
我將與您的命令等效的命令放入循環(huán)中并進行了 100 次迭代，耗時 15 秒.然后我將您對 PNG 文件的所有讀取更改為對 MPC 文件或 Magick Pixel Cache 文件的讀取.這將處理時間縮短到不到 10 秒，即減少了 33%.Magic Pixel Cache 只是一個預(yù)解壓、預(yù)解碼的文件，可以直接讀入內(nèi)存而無需任何 CPU 工作.您可以在目錄更改時預(yù)先創(chuàng)建它們并將它們與 PNG 文件一起存儲.做一個你做的
I put the equivalent of your command into a loop and did 100 iterations and it takes 15 seconds. I then changed all your reads of PNG files into reads of MPC files - or Magick Pixel Cache files. That reduced the processing time to just under 10 seconds, i.e. by 33%. A Magic Pixel Cache is just a pre-decompressed, pre-decoded file that can be read directly into memory without any CPU effort. You could pre-create them whenever your catalogue changes and store them alongside the PNG files. To make one you do
convert image.png image.mpc
你會得到image.mpc和image.cache.然后您只需將代碼更改為如下所示:
and you will get out image.mpc and image.cache. Then you would simply change your code to look like this:
convert -size 600x400 xc:none
( 1.mpc -fill rgb(x,y,z) -colorize 100% ) -composite
( 2.mpc -fill rgb(x,y,z) -colorize 100% ) -composite
( 3.mpc -fill rgb(x,y,z) -colorize 100% ) -composite
( 4.mpc -fill rgb(x,y,z) -colorize 100% ) -composite
( 5.mpc -fill rgb(x,y,z) -colorize 100% ) -composite
( 6.mpc -fill rgb(x,y,z) -colorize 100% ) -composite
result.png
第 3 點
很遺憾，您還沒有回答我的問題，但如果您的資產(chǎn)目錄不是太大，您可以在系統(tǒng)啟動時將其(或上面的 MPC 等效項)放到 RAM 磁盤上.
Unfortunately you haven't answered my questions yet, but if your assets catalogue is not too big, you could put that (or the MPC equivalents above) onto a RAM disk at system startup.
第 4 點
您絕對應(yīng)該并行運行 - 這將產(chǎn)生最大的收益.使用 GNU Parallel 非常簡單 - 此處的示例.
You should definitely run in parallel - that will yield the biggest gains of all. It is very simple with GNU Parallel - example here.
如果您使用的是 REDIS，實際上比這更容易.只需 LPUSH 將您的 MIME 編碼圖像放入一個 REDIS 列表中，如下所示:
If you are using REDIS, it is actually easier than that. Just LPUSH your MIME-encoded images into a REDIS list like this:
#!/usr/bin/perl
################################################################################
# generator.pl <number of images> <image size in bytes>
# Mark Setchell
# Base64 encodes and sends "images" of specified size to REDIS
################################################################################
use strict;
use warnings FATAL => 'all';
use Redis;
use MIME::Base64;
use Time::HiRes qw(time);
my $Debug=0; # set to 1 for debug messages
my $nargs = $#ARGV + 1;
if ($nargs != 2) {
print "Usage: generator.pl <number of images> <image size in bytes>
";
exit 1;
}
my $nimages=$ARGV[0];
my $imsize=$ARGV[1];
# Our "image"
my $image="x"x$imsize;
printf "DEBUG($$): images: $nimages, size: $imsize
" if $Debug;
# Connection to REDIS
my $redis = Redis->new;
my $start=time;
for(my $i=0;$i<$nimages;$i++){
my $encoded=encode_base64($image,'');
$redis->rpush('images'=>$encoded);
print "DEBUG($$): Sending image $i
" if $Debug;
}
my $elapsed=time-$start;
printf "DEBUG($$): Sent $nimages images of $imsize bytes in %.3f seconds, %d images/s
",$elapsed,int($nimages/$elapsed);
然后運行多個工人，他們都坐在那里做大量的工作
and then run multiple workers that all sit there doing BLPOPs of jobs to do
#!/usr/bin/perl
################################################################################
# worker.pl
# Mark Setchell
# Reads "images" from REDIS and uudecodes them as fast as possible
################################################################################
use strict;
use warnings FATAL => 'all';
use Redis;
use MIME::Base64;
use Time::HiRes qw(time);
my $Debug=0; # set to 1 for debug messages
my $timeout=1; # number of seconds to wait for an image
my $i=0;
# Connection to REDIS
my $redis = Redis->new;
my $start=time;
while(1){
#my $encoded=encode_base64($image,'');
my (undef,$encoded)=$redis->blpop('images',$timeout);
last if !defined $encoded;
my $image=decode_base64($encoded);
my $l=length($image);
$i++;
print "DEBUG($$): Received image:$i, $l bytes
" if $Debug;
}
my $elapsed=time-$start-$timeout; # since we waited that long for the last one
printf "DEBUG($$): Received $i images in %.3f seconds, %d images/s
",$elapsed,int($i/$elapsed);
如果我像上面一樣運行一個生成器進程并讓它生成 100,000 張圖像，每個圖像 200kB，并在我的合理規(guī)格 iMac 上用 4 個工作進程讀取它們，需要 59 秒，或者大約 1,700 張圖像/秒可以通過 REDIS.
If I run one generator process as above and have it generate 100,000 images of 200kB each, and read them out with 4 worker processes on my reasonable spec iMac, it takes 59 seconds, or around 1,700 images/s can pass through REDIS.
這篇關(guān)于使用 redis/php-resque 優(yōu)化并發(fā) ImageMagick 請求的文章就介紹到這了，希望我們推薦的答案對大家有所幫助，也希望大家多多支持html5模板網(wǎng)！