Microsoft MPI и Visual Studio

ВНИМАНИЕ, ЭТО НЕДОПИСАННАЯ СТАТЬЯ. Она мо­жет со­дер­жать ка­кие-угод­но не­точ­но­сти.

В од­ной из преды­ду­щих ста­тей я по­дроб­но опи­сал про­цесс уста­нов­ки и на­строй­ки биб­лио­те­ки MPICH 2 (са­мой по­пу­ляр­ной от­кры­той реа­ли­за­ции MPI) в опе­ра­ци­он­ной си­сте­ме Windows, и по­лу­чил мно­же­ство от­зы­вов. К со­жа­ле­нию, лю­ди ча­сто жа­лу­ют­ся на про­бле­мы с на­строй­кой MPICH в этой опе­ра­ци­он­ной си­сте­ме, что не­уди­ви­тель­но: на про­тя­же­нии мно­гих лет Windows яв­ля­лась враж­деб­ной плат­фор­мой для на­уч­ных вы­чис­ле­ний и сво­бод­но­го про­граммно­го обес­пе­че­ния. По­это­му те­перь рас­смот­рим реа­ли­за­цию MPI от са­мой Microsoft.

Мод­ная аб­бре­ви­а­ту­ра HPC рас­шиф­ро­вы­ва­ет­ся, как High-Per­for­man­ce Com­pu­ting — вы­со­ко­про­из­во­ди­тель­ные вы­чис­ле­ния. Под эту от­расль нын­че вы­де­ля­ют­ся боль­шие сред­ства как за ру­бе­жом, так и в Рос­сии

Не­сколь­ко лет на­зад Microsoft осо­зна­ла своё от­ста­ва­ние в ни­ше вы­со­ко­про­из­во­ди­тель­ных вы­чис­ле­ний, и в 2006 го­ду вы­пу­сти­ла Windows Compute Cluster Server 2003, име­ю­щую на­бор до­пол­ни­тель­ных API-функ­ций для на­строй­ки и ра­бо­ты Windows-ма­ши­ны в ка­че­стве уз­ла вы­чис­ли­тель­но­го кла­сте­ра, а так­же соб­ствен­ную реа­ли­за­цию стан­дар­та MPI — MS MPI.

За­тем был со­здан Windows HPC Server 2008, имею­щий все ат­ри­бу­ты HPC-си­сте­мы, в част­но­сти:

  • пря­мой до­ступ из вы­чис­ли­тель­ной про­грам­мы к се­те­во­му обо­ру­до­ва­нию;
  • под­си­сте­му раз­вёр­ты­ва­ния и управ­ле­ния кла­сте­ром;
  • ме­не­джер вы­чис­ли­тель­ных за­да­ний;
  • под­держ­ку раз­лич­ных стан­дар­тов, при­ня­тых в ми­ре HPC;
  • оче­ред­ную вер­сию MS MPI, о ко­то­рой мы и по­го­во­рим.

Что­бы опро­бо­вать MS MPI в дей­ствии, во­все не обя­за­тель­но раз­вёр­ты­вать кла­стер под управ­ле­ни­ем Windows HPC Server; до­ста­точ­но уста­но­вить се­бе не­боль­шой по объ­ё­му па­кет HPC Pack 2008 SDK, ко­то­рый ра­бо­та­ет под опе­ра­ци­он­ны­ми си­сте­ма­ми Windows Server 2003, Windows Server 2008, Windows Vista и Windows XP. Же­ла­тель­но иметь не­сколь­ко ком­пью­те­ров, со­еди­нён­ных ги­га­бит­ной се­тью, но сой­дёт и один ком­пью­тер с мно­го­ядер­ным про­цес­со­ром (очень не ре­ко­мен­дую ис­поль­зо­вать сеть Wi-Fi да­же для учеб­ных це­лей).

Уста­нов­ка HPC Pack 2008 SDK

Уста­нов­ка HPC Pack SDK по­хо­жа на уста­нов­ку дру­гих про­грамм­ных про­дук­тов Microsoft:

Я не уве­рен, что вер­сии раз­ной «бит­но­сти» бу­дут пра­виль­но свя­зы­вать­ся друг с дру­гом по се­ти

  1. ска­чай­те 32-х бит­ную или 64-х бит­ную вер­сию, в со­от­вет­ствии с вер­си­ей ва­шей опе­ра­ци­он­ной си­сте­мы;
  2. за­пу­сти­те ска­чан­ный ин­стал­ля­тор;
  3. при­ми­те ли­цен­зи­он­ное со­гла­ше­ние;
  4. на­жи­май­те кноп­ку «Next» до тех пор, по­ка она не пре­вра­тит­ся в кноп­ку «Finish»;
  5. ска­чай­те Fix и уста­но­ви­те его то­же;

В от­ли­чие от MPICH, на­страи­вать HPC Pack 2008 SDK не нуж­но.

На­строй­ка Visual Studio

Те­перь нам нуж­но на­стро­ить Visual Studio для ра­бо­ты с вновь про­ин­стал­ли­ро­ван­ной биб­лио­те­кой. Ес­ли у вас за­пу­ще­на Visual Studio, то на вся­кий слу­чай пе­ре­за­пу­сти­те её по­сле уста­нов­ки HPC Pack SDK. На­жми­те Tools → Options, и до­бавь­те путь к за­го­ло­воч­ным фай­лам в Projects and Solutions → VC++ Directories → Show directories for: Include files. Пред­по­ла­га­ем, что вы уста­но­ви­ли HPC Pack SDK в стан­дарт­ное ме­сто:

До­бав­ля­ем путь к за­го­ло­воч­ным фай­лам MS MPI
До­бав­ля­ем путь к за­го­ло­воч­ным фай­лам MS MPI

Ри­су­нок 1. До­бав­ля­ем путь к за­го­ло­воч­ным фай­лам MS MPI

Ана­ло­гич­но до­бавь­те ди­рек­то­рию с биб­лио­теч­ны­ми фай­ла­ми. Ес­ли у вас 32-х бит­ная опе­ра­ци­он­ная си­сте­ма, то путь окан­чи­ва­ет­ся на i386, ес­ли же си­сте­ма 64-х бит­ная, то путь окан­чи­ва­ет­ся на amd64:

До­бав­ля­ем путь к биб­лио­теч­ным фай­лам MS MPI
До­бав­ля­ем путь к биб­лио­теч­ным фай­лам MS MPI

Ри­су­нок 2. До­бав­ля­ем путь к биб­лио­теч­ным фай­лам MS MPI

Со­зда­ние про­стей­шей про­грам­мы

Со­здай­те кон­соль­ный про­ект так, как по­ка­за­но на ри­сун­ках 2–4 в ста­тье об OpenMP. Бу­дем да­лее счи­тать, что вы за­да­ли имя про­ек­та «MpiTest». На­жми­те Project → MpiTest Options, и до­бавь­те msmpi.lib в спи­сок до­пол­ни­тель­ных фай­лов для ком­по­нов­ки:

До­бав­ле­ние msmpi.lib к про­грам­ме

Ри­су­нок 3. До­бав­ле­ние msmpi.lib к про­грам­ме

Вве­ди­те ис­ход­ный код про­грам­мы. Бу­дем вы­пол­нять те­сты на сле­дую­щем про­стей­шем ко­де:

#include <mpi.h>
#include <iostream>
using namespace std;

int main(int argc, char **argv)
{
    MPI_Init(&argc, &argv);
   
    int rank, size;
   
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    if( rank )
    {
        char buf[] = "Hello!";
        MPI_Send(buf, sizeof(buf), MPI_CHAR, 0, 0, MPI_COMM_WORLD);
    } else {
        cout << "Process 0 started" << endl;
        for(int i(1); i<size; ++i)
        {
            MPI_Status s;
            MPI_Probe(MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &s);
            int count;
            MPI_Get_count(&s, MPI_CHAR, &count);

            char *buf = new char[count];
            MPI_Recv(buf, count, MPI_CHAR,
                     MPI_ANY_SOURCE, MPI_ANY_TAG, MPI_COMM_WORLD, &s);
            cout << "Message from process " << s.MPI_SOURCE << ": "
                 << buf << endl;
            delete[] buf;
        }
        cout << "Done." << endl;
    }

    MPI_Finalize();
    return 0;
}

Ли­стинг 1. Про­стей­шая про­грам­ма, ис­поль­зую­щая MPI

На­жми­те Debug → Start Without Debugging. Ес­ли Visual Studio спро­сит вас, хо­ти­те ли от­ком­пи­ли­ро­вать про­грам­му, на­жми­те «Yes». По­сле это­го про­грам­ма от­ком­пи­ли­ру­ет­ся и за­пу­стит­ся. Ес­ли у вас вклю­чён брэнд­мау­эр, то он, воз­мож­но, спро­сит, хо­ти­те ли вы раз­ре­шить про­грам­ме MpiTest.exe до­ступ в сеть. На­жми­те «Раз­бло­ки­ро­вать». По­сле это­го вы долж­ны уви­деть на экра­не кон­соль­ное ок­но с ре­зуль­та­том ра­бо­ты:

За­пуск од­но­го про­цес­са

Ри­су­нок 4. За­пуск од­но­го про­цес­са

За­пуск про­грам­мы на од­ном ком­пью­те­ре

Для бо­лее пол­но­го по­ни­ма­ния ма­те­ри­ла ре­ко­мен­дую озна­ко­мить­ся с прин­ци­па­ми ра­бо­ты MPICH, ко­то­рые я опи­сал в дру­гой ста­тье. В этом пла­не MS MPI по­чти не от­ли­ча­ет­ся от MPICH

Для за­пус­ка MPI-про­грамм в ком­плек­те MS MPI идут про­грам­мы mpiexec.exe и smpd.exe, путь к ко­то­рым при уста­нов­ке HPC Pack SDK до­ба­вил­ся в пе­ре­мен­ную окру­же­ния PATH. Это озна­ча­ет, что вы мо­же­те вы­зы­вать mpiexec.exe из ко­манд­ной стро­ки, на­хо­дясь в лю­бом ме­сте фай­ло­вой си­сте­мы.

Для за­пус­ка на­шей про­грам­мы на од­ном ком­пью­те­ре до­ста­точ­но толь­ко про­грам­мы mpiexec.exe. Она за­пу­стит тре­бу­е­мое чис­ло эк­зем­пля­ров MPI-про­грам­мы и снаб­дит их ин­фор­ма­ци­ей для свя­зи друг с дру­гом. За­пу­сти­те ко­манд­ную стро­ку (Пуск → Про­грам­мы → Стан­дарт­ные → Ко­манд­ная стро­ка), и пе­рей­ди­те в пап­ку с ис­пол­няе­мым фай­лом ва­шей про­грам­мы с по­мо­щью ко­ман­ды cd, за­тем за­пу­сти­те про­грам­му с по­мо­щью mpiexec:

За­пуск пя­ти про­цес­сов на од­ном ком­пью­те­ре
За­пуск пя­ти про­цес­сов на од­ном ком­пью­те­ре

Ри­су­нок 5. За­пуск пя­ти про­цес­сов на од­ном ком­пью­те­ре

Клю­чи -n 5 у про­грам­мы mpiexec го­во­рят о том, что нуж­но за­пу­стить 5 про­цес­сов на те­ку­щем ком­пью­те­ре. Спи­сок до­пу­сти­мых клю­чей мож­но по­лу­чить по ко­ман­де mpiexec -help2. Не удив­ляй­тесь, что со­об­ще­ния от про­цес­сов 1–4 пе­ре­ме­ша­ны: про­цес­сы во­все не обя­за­ны за­пус­кать­ся по оче­ре­ди и ра­бо­тать с оди­на­ко­вой ско­ро­стью.

За­пуск про­грам­мы на не­сколь­ких ком­пью­те­рах

[До­пи­шу поз­же...]

P.S.

Са­мая пол­ная ин­фор­ма­ция о со­зда­нии MPI-про­грамм на­хо­дит­ся в спе­цифи­ка­ции MPI. Ре­ко­мен­дую на­чать изу­че­ние с вер­сии 1.3.

130 отзывов на запись «Microsoft MPI и Visual Studio»

Спа­си­бо за ста­тью, по­мог­ла. А как сде­лать так, что­бы из VS 2008 за­пус­кал­ся не толь­ко про­цесс ма­стер, но и не­сколь­ко дру­гих про­цес­сов, ко­т­рые бу­дут об­ра­ба­ты­вать дан­ные.
Спа­си­бо за ста­тью. по­лез­ная. толь­ко вот уме воз­ни­ка­ет про­бле­ма: ко­гда за­пус­каю из VisualStudio или на пря­мую, то всё нор­маль­но — один за­пу­ще­ный про­цесс. А вот ес­ли за­пус­ка­ешь не­сколь­ко про­цес­сов (да­же ес­ли ука­зать ко­ли­че­ство про­цес­сов 1) по сред­ством mpiexec. то вы­ска­ки­ва­ет ошиб­ка: smpd.exe — Точ­ка вхо­да не най­де­на Точ­ка вхо­да в про­це­ду­ру GETLogicalProcessorInformation не най­де­на в биб­лио­те­ке DLL KERNEL32.dll
Ка­кая у вас вер­сия Windows? Об­нов­ля­ли её?
XP SP3 а что это мо­жет по­вли­ять? у ме­ня про­сто по­став­лен transformationpack MacOs но это вро­де толь­ко ви­зуа­ли­за­ция, не долж­но за­де­вать та­кие биб­лио­те­ки.
Это очень стран­но, ибо в SP3 функ­ция GetLogicalProcessorInformation име­ет­ся (до это­го её не бы­ло). Мо­жет, кри­во уста­нов­лен Service Pack? По­про­буй­те за­но­во уста­но­вить тре­тий Service Pack на ва­шу си­сте­му, ли­бо пе­ре­уста­но­ви­те си­сте­му це­ли­ком. Воз­мож­но так­же, что Transformation Pack что-то по­вре­дил.
спа­си­бо. по­про­бую
да дей­стви­тель­но пе­ре­уста­нов­ка SP3 по­мог­ла, толь­ко вот Пак осов­ский сле­тел. спа­си­бо
здрав­ствуй­те, сде­лал все как ука­за­но вы­ше, при за­пус­ке ис­ход­но­го ко­да про­грам­мы, вы­да­ет ошиб­ку error 1083: Cannot open include file: ‘mpi.h’ в чем мо­жет быть про­бле­ма? де­лал все по ин­ст­рук­ции
разо­брал­ся
при­вет у ме­ня та­кая же про­бле­ма ты как разо­брал­ся от­веть плз
Zdravstvui kak razobralsia? podskaji!
За­пуск про­грам­мы на не­сколь­ких ком­пью­те­рах
[До­пи­шу поз­же...] Са­мо­го нуж­но и нет!
Под­ска­жи­те, что по­чи­тать?
Как мо­жет по­вли­ять на ра­бо­ту Microsoft MPI уста­нов­лен­ный и на­стро­ен­ный для ра­бо­ты с VS MPICH?
Спа­си­бо боль­шое за ста­тью, очень по­мог­ла. Но бы­ла про­бле­ма при до­бав­ле­нии ди­рек­то­рии с биб­лио­теч­ны­ми фай­ла­ми. У ме­ня Windows 7 x64, но при до­бав­ле­нии amd64 вы­да­ва­ло ошиб­ку: «error LNK2019: ссыл­ка на не­раз­ре­шен­ный внеш­ний сим­вол», как бы не мог­ло най­ти функ­ции. Вы­ле­чил за­ме­ной amd64 на i386.
За­ме­нил amd64 на i386, един­ствен­ное что ре­ши­ло ана­ло­гич­ную про­бле­му.
Спа­си­бо за ком­мент, очень по­мог.
Огром­ное спа­си­бо за ста­тью! Всё очень по­дроб­но и кри­сталь­но яс­но. Бу­ду очень рад, ес­ли раз­дел «За­пуск про­грам­мы на не­сколь­ких ком­пью­те­рах» бу­дет до­пи­сан.
Ещё раз спа­си­бо!
Кто-ни­будь с Фор­тра­ном де­ло име­ет? а то вы­да­ет 20 оши­бок по ти­пу та­кой:
1>Klaus_3d.obj : error LNK2019: ссыл­ка на не­раз­ре­шен­ный внеш­ний сим­вол MPI_INIT в функ­ции MAIN__
файл msmpi.lib про­пи­сан
«ссыл­ка на не­раз­ре­шен­ный внеш­ний сим­вол MPI_INIT в функ­ции MAIN__» и по­доб­ные ошиб­ки го­во­рят о том, что за­бы­ли либ-файл под­клю­чить.чи­тай­те вни­ма­тель­но этот man «Со­зда­ние про­стей­шей про­грам­мы
Со­здай­те кон­соль­ный про­ект так, как по­ка­за­но на ри­сун­ках 2–4 в ста­тье об OpenMP. Бу­дем да­лее счи­тать, что вы за­да­ли имя про­ек­та «MpiTest». На­жми­те Project → MpiTest Options, и до­бавь­те msmpi.lib в спи­сок до­пол­ни­тель­ных фай­лов для ком­по­нов­ки:» «Cannot open include file: ‘mpi.h’»
ско­рее все­го из­за то­го, что не про­пи­са­ли или не­пра­виль­но про­пи­са­ли ди­рек­то­рию для ин­клу­дов. бы­ла ещё 1 про­бле­ма при за­пус­ке из кон­со­ли mpiexec.exe -n 5 mpi.exe ру­гал­ся на то, что не мо­жет за­пу­стить smpd.exe. пе­ре­за­груз­ка + по­ме­стил всё из пап­ки C:Program FilesMicrosoft HPC Pack 2008 SDKBin в пап­ку Debug(Release) про­ек­та и за­ра­бо­та­ло
Да нет, msmpi.lib про­пи­сан!
А по­че­му та­кое вы­ле­за­ет, не знаю!
И ко­гда же и где же ту­то­ри­ал про то как на­стро­ить кла­стер из 2х вин­до­вых ма­шин?=)
Для поль­зо­ва­те­лей Фор­тра­на. пи­шем в свой­ствах про­ек­та в Linker => Input => Additional Dependencies сле­дую­щие 2 биб­лио­те­ки: msmpi.lib msmpifmc.lib
очень ну­жен ту­то­ри­ал по за­пус­ку про­ги на 2ух(3х и бо­лее) windows ком­пах.
Есть про­бле­ма. Ком­пи­ля­ция про­грам­мы про­хо­дит нор­маль­но. Но ко­гда я хо­чу за­пу­стить exe фалй че­рез ко­манд­ную стро­ку, мне вы­би­ва­ет иеро­гли­фы.
Вот что пи­шет:
Ско­пи­руй­те сам текст сю­да, ина­че не смо­гу про­честь. Для это­го в в ме­ню ок­на на­жми­те из­ме­нить → по­ме­тить, вы­де­ли­те текст, и на­жми­те Enter — он ско­пи­ру­ет­ся в бу­фер об­ме­на.
Microsoft Windows [Version 6.1.7600]
(c) Кор­по­ра­ция Май­к­ро­софт (Microsoft Corp.), 2009. Все пра­ва за­щи­ще­ны. C:UsersПа­вел>cd «d:\\mpidebug» C:UsersПа­вел>mpiexec -n 5 mpi.exe Aborting: failed to launch ‘mpi.exe’ on ╧рт­хы-╧╩
Error (2) ═х єфрхЄё  эрщЄш єър­чрээ√щ Їрщы. C:UsersПа­вел>
На­пи­са­но: Aborting: failed to launch \'mpi.exe\' on Па­вел-gi
Error (2) Не уда­ет­ся най­ти ука­зан­ный файл.
Ви­ди­мо, файл mpi.exe не на­хо­дит­ся в пап­ке C:UsersПа­вел. Пе­рей­ди­те в нуж­ную пап­ку и от­ту­да за­пус­кай­те.
Боль­шое спа­си­бо! Все по­лу­чи­лось! Я не ду­мал да­же что мо­жет быть про­бле­ма в пу­ти. На­ско­льок я по­ни­маю что в ОС в Pach — про­пи­са­ный путь к exe и мож­но с лю­бой точ­ки за­пус­кать про­ком­пел­ли­ро­ва­ный файл. Спа­си­бо! Вы мне очень по­мог­ли!)))
Есть про­бле­ма. Ком­пи­ля­ция про­грам­мы про­хо­дит нор­маль­но. Но ко­гда я хо­чу за­пу­стить exe файл че­рез ко­манд­ную стро­ку все вис­нет C:Documents and SettingsAdmin>mpiexec -timeout 3 -n 5 1.exe mpiexec terminated job due to 3 second timeout.
Вот та­кая ошиб­ка
http://i079.radikal.ru/1203/5f/79b05d97377a.jpg
хо­ро­шая ста­тья, про­дол­жай­те пи­сать!
От­лич­ная ста­тья! Под­ска­жи­те по­жа­луй­ста, есть ли спо­соб про­во­дить от­лад­ку в VS 2010 Fortran при­ло­же­ния с MPI, так что­бы не­по­сред­ствен­но в от­лад­чи­ке за­пус­ка­лось не­сколь­ко про­цес­сов, и break points сра­ба­ты­ва­ли на всех про­цес­сах? в VS 2008 мож­но бы­ло вы­брать тип от­лад­чи­ка MPI Cluster Debugger для Fortran, а в VS 2010 так уже не по­лу­ча­ет­ся..
Един­ствен­ный спо­соб, ко­то­рый я на­шел — при­со­еди­нить от­лад­чик к про­цес­сам за­пу­щен­ным из mpiexec. Но это за­ни­ма­ет вре­мя, а как при­со­еди­нить от­лад­чик как толь­ко за­пус­ка­ет­ся про­цесс я не знаю.
http://msdn.microsoft.com/ru-ru/library/ee441265.aspx
Боль­шое спа­си­бо за ста­тью!
За­ме­ча­тель­ная ста­тья! Боль­шое спа­си­бо! Един­ствен­ное, я очень дол­го ду­ма­ла, ка­кой же HPC Pack нуж­но ска­чать. В кон­це-кон­цов ска­ча­ла HPC Pack 2008 R2 MS-MPI Redistributable Package with Service Pack 4 (http://www.microsoft.com/en-us/download/details.aspx?id=14737). Все по­лу­чи­лось!
Все еще ждем про­дол­же­ния!
Спа­си­бо за ста­тью. Вот крат­кая ин­фор­ма­ция по ис­поль­зо­ва­нию MPI.NET на не­сколь­ких уз­лах: The following is a way to run 2 MPI.NET PI.exe on 2 machines (could be XP) as a cluster 1) on each machine, start smpd.exe as: smpd -d 2) copy PI.exe to each machine’s c:tests directory 3) go to one of the machines, run: mpiexec -hosts 2 machine1 1 machine2 1 c:testsPI.exe this will run PI.exe on 2 machines, with each machine running 1 process. The -hosts option’s syntax is: -hosts 2 machine1 N machine2 M: where to run N processes on machine1 and M processes on machine2. If you don’t want to copy PI.exe to each machine, it is a little bit complex. the basic process is a) create a share on some machine. e.g. \\machine1testshare, copy PI.exe there. b) use .Net caspol.exe to set up the security policy on the above share. c) go to one of the machines, run: mpiexec -hosts 2 machine1 1 machine2 1 \\machine1testsharePI.exe где machine1 — ip-ад­рес (hostname) пе­ро­во­го ком­пью­те­ра и т.д.
Все от­лич­но опи­са­но. Спа­си­бо за хо­ро­шее из­ло­же­ние ин­фор­ма­ции.
Боль­шое спа­си­бо за по­дроб­ное опи­са­ние! Как и Еле­на вы­брал дис­три­бу­тив HPC Pack 2008 R2 MS-MPI Redistributable Package with Service Pack 4 (http://www.microsoft.com/en-us/download/details.aspx?id=14737). Те­сто­вый при­мер сра­зу за­ра­бо­тал. У ме­ня ПО Windows 7-32, Visual Studio 2008
Ад­рес HPC Pack… не­жи­вой! Мож­но уточ­нить, от­ку­да брать? Во­прос -
за­пус­ка­ли не­сколь­ко про­цес­сов на од­ной ма­ши­не или на не­сколь­ких
по се­ти? У ме­ня про­бле­ма — хо­чу со­еди­нить не­сколь­ко ма­шин ком­пью­тер­но­го
клас­са на Win’7 Prof 64 бит, но встре­чал мно­го ин­фор­ма­ции, что с
Win’7 c МPICH про­бле­мы. Ка­кой ва­ри­ант MPICH взять для мо­ей це­ли?
Ес­ли mpi не под­ру­ба­ет­ся, то став­те Microsoft Compute Cluster Pack SDK за­ме­сто HPC Pack.
По­ту про­пи­сы­вай­те в не­го.
спа­си­бо боль­шое, по­мог­ло.)
Hello!
Hello!
Hello!
Hello!
Hello!
Hello!
Hello!
Every weekend i used to pay a visit this website, for the reason that i wish for enjoyment, for the reason
It’s perfect time to make some plans for the long run and it is time to be happy. I have learn this publish and if I may I want to suggest you few attentiongrabbing things or advice. Maybe you could write next articles regarding this article. I desire to read even more things about it!
griseofulvin psoriasis
Hi there, I found your site by way of Google while searching for a comparable matter, your web site got here up, it looks good. I have bookmarked it in my google bookmarks.
what are some really good and popular websites for blogging? ?? .
Heya im for the first time here. I discovered this board and I to uncover It truly helpful &amp it helped me out a whole lot. I hope to supply something back and aid other people such as you helped me.
My brother suggested I might like this blog. He was entirely right.
This post actually made my day. You can not imagine simply how much
time I had spent for this information! Thanks!
Hello, I enjoy reading all of your article. I like to write a little comment to support you.
If some one wishes expert view about running a blog then i
recommend him/her to pay a visit this webpage, Keep up the fastidious job.
I do not even know how I ended up here, but I thought this post was great.
I don’t know who you are but definitely you are
going to a famous blogger if you aren’t already ;
) Cheers!
A fascinating discussion is definitely worth comment.
There’s no doubt that that you need to publish more on this
topic, it may not be a taboo matter but generally people do not speak about such topics. To the next! Kind regards!!
My programmer is trying to convince me to move to .net from
PHP. I have always disliked the idea because of the expenses. But he’s tryiong none the less. I’ve been using WordPress on a variety
of websites for about a year and am concerned about switching to another
platform. I have heard great things about
blogengine.net. Is there a way I can import all my wordpress posts into it?
Any help would be really appreciated!
Its like you read my mind! You seem to know a lot about this, like you wrote the
book in it or something. I think that you can do with some
pics to drive the message home a little bit, but instead of that, this is great blog.
An excellent read. I will certainly be back.
Hi to every one, as I am in fact keen of reading this weblog’s post
to be updated on a regular basis. It carries pleasant data.
bookmarked!!, I love your site!
Very energetic blog, I loved that bit. Will there be a part 2?
Excellent goods from you, man. I’ve understand your
stuff previous to and you are just extremely great. I actually like what you’ve acquired here,
certainly like what you are saying and the way in which you say it.
You make it entertaining and you still take care of to keep it wise.
I can’t wait to read much more from you. This is really a
wonderful website.
I have been surfing on-line more than 3 hours as of late, but I never discovered any fascinating article like yours.
It’s beautiful value sufficient for me. In my opinion, if all website owners and bloggers
made good content material as you did, the net will probably be much more useful than ever before.
I enjoy what you guys are usually up too. This type of clever work and reporting!
Keep up the terrific works guys I’ve included you
guys to my own blogroll.
Everything is very open with a really clear clarification of the issues.
It was definitely informative. Your site is useful.
Thanks for sharing!
Everyone loves it when people get together and share
thoughts. Great blog, stick with it!
Thanks a bunch for sharing this with all of us you actually
realize what you’re talking approximately! Bookmarked.
Kindly also talk over with my web site =). We can have a hyperlink alternate arrangement between us
Hey there I am so excited I found your web site,
I really found you by accident, while I was researching on Digg for something else,
Regardless I am here now and would just like to say
many thanks for a remarkable post and a all round interesting blog (I also love the theme/design), I don’t have
time to look over it all at the minute but I have bookmarked it and also added your RSS
feeds, so when I have time I will be back to read much more, Please do keep up
the awesome job.
I’d like to find out more? I’d love to find out more details.
Undeniably believe that which you stated. Your favorite reason appeared to be on the web the simplest thing to be aware
of. I say to you, I certainly get annoyed while people
think about worries that they plainly do not know
about. You managed to hit the nail upon the top as well as defined out the whole thing without having side effect , people could take
a signal. Will probably be back to get more. Thanks
It’s a shame you don’t have a donate button! I’d without a doubt
donate to this excellent blog! I guess for now i’ll settle for book-marking and adding your RSS feed to my Google account.
I look forward to fresh updates and will talk about
this blog with my Facebook group. Chat soon!
Hi there to every , as I am really keen of reading this blog’s post to be updated on a regular basis.
It consists of good information.
I always used to read piece of writing in news papers but now as I am a user of net thus from now
I am using net for content, thanks to web.
Undeniably believe that which you stated. Your favorite
reason seemed to be on the web the easiest thing
to be aware of. I say to you, I definitely get annoyed while people consider
worries that they just don’t know about. You managed to hit the nail
upon the top and defined out the whole thing without having side effect , people could
take a signal. Will probably be back to get more. Thanks
Yes! Finally something about tinder dating site.
Greetings from Carolina! I’m bored at work so I decided to check out your site on my iphone during
lunch break. I love the knowledge you provide here and can’t wait to take a look when I get home. I’m amazed at how fast your blog loaded on my mobile ..
I’m not even using WIFI, just 3G .. Anyhow, good blog!
If some one needs expert view concerning running a blog afterward i suggest him/her to pay a quick visit this website,
Keep up the pleasant job. Review my web-site silk lash extensions perth (Adrian)
When I initially commented I clicked the «Notify me when new comments are added» checkbox and now each time
a comment is added I get several e-mails with the same comment.
Is there any way you can remove people from that service?
Thanks! Here is my website silk lash extensions london
Excellent post. I was checking constantly this
blog and I am impressed! Extremely useful information specifically the last part I care for
such info much. I was seeking this certain info for a very long time. Thank you and best of luck.
Informative article, totally what I was looking for.
Great info. Lucky me I ran across your website by accident (stumbleupon).
I have book marked it for later! Also visit my blog post: mink lashes kit
Hey there! This is kind of off topic but I need some advice
from an established blog. Is it very hard to set up your own blog? I’m not very techincal but I can figure things out pretty fast.
I’m thinking about making my own but I’m not sure where to start.
Do you have any tips or suggestions? Thanks my page :: unique episode (http://5.83.128.199)
Magnificent goods from you, man. I have understand your stuff previous
to and you are just too fantastic. I really like what you have acquired here, certainly like
what you are stating and the way in which you say
it. You make it entertaining and you still care for to keep it smart.
I can’t wait to read much more from you. This is really a wonderful
website. my web page; play video games (Erma)
There’s definately a great deal to learn about this subject.
I like all the points you’ve made. Feel free to surf to my blog post … cs go skins trade
Hi! I just wanted to ask if you ever have any issues with hackers? My last blog (wordpress) was hacked and I ended up losing months of hard work due to
no backup. Do you have any solutions to stop hackers? Feel free to visit my page csgo skins free no survey — yrs-jameh.com
-
Hey very interesting blog! my web blog … madden mobile coins for sale reddit (Kelley)
My family all the time say that I am killing my time here
at web, however I know I am getting experience everyday by reading
such pleasant articles or reviews. Feel free to surf to my web-site — madden mobile free coins no survey no download
What’s up all, here every one is sharing such knowledge, therefore it’s nice to read
this webpage, and I used to pay a visit this blog daily. Review my web-site — mink eyelash extensions kitchener
Howdy very cool site!! Guy .. Excellent .. Superb ..
I will bookmark your site and take the feeds also?
I am satisfied to seek out a lot of helpful information here within the
submit, we want work out more strategies in this regard, thank you for
sharing. . . . . . my site — mink strip lashes uk (http://yrs-jameh.com/content/how-order-and-wear-fake-eye-lashes-6-steps)
That is a very good tip especially to those fresh to the blogosphere.
Simple but very accurate info… Thanks for sharing this one.
A must read article! Check out my web-site: fifa coins zone
I love what you guys are usually up too. This type of clever work and exposure!
Keep up the very good works guys I’ve added you guys to my blogroll. Here is my web-site … cs go skins trade up
Does your blog have a contact page? I’m having trouble locating it but, I’d like to shoot you an e-mail.
I’ve got some recommendations for your blog you might be interested in hearing.
Either way, great website and I look forward to seeing it
develop over time. Feel free to surf to my weblog — cs go trade skins for money
I think this is one of the most important information for
me. And i am glad reading your article. But wanna remark on some general things, The web site style is ideal, the articles is really
excellent : D. Good job, cheers my weblog: fifa 17 coins for sale cheap
This paragraph is truly a good one it helps new web users, who are wishing for blogging. Here is my web-site :: diamond silk lashes wholesale
It’s the best time to make some plans for the future and
it is time to be happy. I have read this post and if I could I wish to suggest you few interesting things
or suggestions. Perhaps you could write next articles referring to this article.
I desire to read even more things about it!
Hi there are using WordPress for your site platform? I’m new to the blog world but I’m trying to get started and set up my own. Do
you need any html coding knowledge to make your own blog?
Any help would be greatly appreciated! my web page: fifa ultimate team coins reddit
I every time used to study article in news papers but now as I am
a user of internet so from now I am using net for
content, thanks to web. Feel free to surf to my blog post csgo skins free spin
I am sure this piece of writing has touched
all the internet viewers, its really really good article on building up new web site. my website … silk lashes review
You can definitely see your enthusiasm within the
work you write. The world hopes for more passionate writers such as you who are not afraid to say how they believe.
Always follow your heart. my homepage: cs go trade skins for money -
Valorie,
It’s a shame you don’t have a donate button! I’d without a doubt donate
to this fantastic blog! I suppose for now i’ll settle for book-marking and adding your RSS feed to my
Google account. I look forward to fresh updates and will share this website with my
Facebook group. Talk soon! My web site fifa coins for sale
I feel this is one of the most important information for me. And i am satisfied studying your article. But wanna statement on some common issues, The site taste is ideal, the articles is really excellent : D.
Just right job, cheers My site; mink lashes amazon
magnificent points altogether, you just gained a new reader. What might you recommend about your submit that you just made some days ago?
Any certain? my web blog :: fifa coins ps4
What i do not realize is in reality how you are now not really much more well-preferred than you might
be right now. You’re very intelligent. You realize thus considerably on the
subject of this subject, produced me individually believe it from so
many various angles. Its like women and men don’t seem to be fascinated except it’s one thing to accomplish with Girl gaga!
Your individual stuffs nice. Always deal with it up! Review my blog post :: cs go skins for sale ebay
My partner and I absolutely love your blog and find many of your
post’s to be exactly I’m looking for. Would you offer guest writers to write content for you personally?
I wouldn’t mind publishing a post or elaborating on some of the subjects you write concerning here.
Again, awesome web log!
When someone writes an piece of writing he/she retains
the image of a user in his/her mind that how a user can be aware of it.
So that’s why this post is amazing. Thanks!
Howdy! I could have sworn I’ve been to this blog before
but after reading through some of the post I realized it’s new to me.
Anyhow, I’m definitely glad I found it and I’ll be bookmarking
and checking back frequently!
Pretty great post. I simply stumbled upon your weblog and wanted to mention that I have really loved surfing around your weblog posts.
In any case I will be subscribing on your feed and I am hoping you write once more very
soon! Here is my website mink eyelash extensions near me
I believe that is one of the such a lot significant information for me.
And i am happy reading your article. However want to observation on few
common issues, The site taste is ideal, the
articles is truly great : D. Good activity, cheers Feel free to visit my blog: fifa coins free ps4
I think that is among the such a lot important info for me.
And i am glad reading your article. However wanna
commentary on some general things, The website style is great, the articles is truly excellent :
D. Just right task, cheers My site … cs go skins steam workshop
excellent issues altogether, you just won a emblem new reader. What would you suggest about your submit that
you made some days in the past? Any positive?
Good day! Do you know if they make any plugins to protect against hackers?
I’m kinda paranoid about losing everything I’ve worked hard
on. Any suggestions?
naturally like your web site however you have to take a look at the spelling on several of your posts.
Many of them are rife with spelling issues and I in finding
it very troublesome to tell the reality on the other hand I’ll certainly come again again.
I am not sure where you’re getting your information, but good topic. I needs to spend some time learning much more or understanding more. Thanks for fantastic info I was looking for this information for my mission.
I like reading through a post that will make people think.
Also, many thanks for permitting me to comment!
Write more, thats all I have to say. Literally, it seems as
though you relied on the video to make your point.
You definitely know what youre talking about, why waste your intelligence on just posting videos to your site when you could be giving us something
informative to read?
It’s in point of fact a great and useful piece of information. I’m satisfied that
you just shared this useful information with us.
Please stay us informed like this. Thanks for sharing. Here is my web site selling cs go skins for cash; Michel,
I don’t know if it’s just me or if everybody else experiencing problems with your blog.
It looks like some of the written text in your posts are running off the screen.
Can someone else please comment and let me know if this
is happening to them as well? This might be a issue with my web
browser because I’ve had this happen before. Kudos my homepage … madden mobile unlimited coins cheat
It’s nearly impossible to find experienced people
in this particular subject, however, you sound like you know what
you’re talking about! Thanks my webpage — cs go howl skin for sale (Shaunte)
Great blog right here! Additionally your web site loads up fast!
What web host are you the use of? Can I am getting your associate hyperlink in your host?
I desire my web site loaded up as quickly as yours lol Feel free to visit my weblog mink lashes kit (http://www.ansonhealth.org/)
Hello, I enjoy reading through your article. I wanted to write a little comment to support you. my web site … madden mobile unlimited coins no survey no download (aieetsc.com)
Hey there! I know this is somewhat off topic but I was wondering
if you knew where I could get a captcha plugin for my comment form?
I’m using the same blog platform as yours and I’m having difficulty finding one?
Thanks a lot!
Hello, i feel that i noticed you visited my blog
thus i got here to go back the want?.I’m attempting to to find
things to enhance my site!I assume its good enough to
make use of a few of your ideas!!
Great post but I was wanting to know if you could write a litte more
on this subject? I’d be very grateful if you could elaborate a little bit more. Many thanks! Feel free to surf to my blog post :: cs go skins
trade up contract — brk-burglengenfeld.de -
Hi there this is somewhat of off topic but I
was wondering if blogs use WYSIWYG editors or if you have to manually
code with HTML. I’m starting a blog soon but have
no coding know-how so I wanted to get guidance from someone with experience.
Any help would be greatly appreciated! Here is my page — silk lash extensions perth
Great info. Lucky me I discovered your site by accident
(stumbleupon). I’ve book marked it for later! my web-site — madden mobile coins buy reddit
Just desire to say your article is as surprising. The clarity on your
put up is just excellent and that i can suppose you’re knowledgeable in this
subject. Fine along with your permission allow me to snatch your
feed to stay up to date with coming near near post. Thank you a million and please continue the
enjoyable work. Feel free to visit my page … mink strip lashes canada

Оставить отзыв

Жёлтые поля обязательны к заполнению

   

Можете использовать теги <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang=""> <div class=""> <span class=""> <br>