Summing up today

I already was able to produce some nice graphs, but still depending a lot on manual editing, instead of auto-generation. Problem is, I first need to output to CVS, and take care of all the strange things in the spreadsheet.

I was looking at the possibility of using a graphs tool, and I found it, but it's not immediate to make the transformation, and particularly make some transformations all around. maybe Excel 2003 has a better support? The aim of the project is *not* transforming.

Anyway, today, produced:
under /GMAN/tests/shell:
> gmanNcTestRun.sh #creates batch messages and sends them. configurable inside (report to previous posts)
> runSomeTests. good ones were : timedResults{21,23,25}.log[.xml] this were already outputed to csvTestResults.csv, and edited in csvTestResults.sxc I might delete all the rest? :S
> test.xslt. Transformation of the current output to a different one (according to the software downloaded: http://www.maani.us/xml_charts/index.php?menu=Tutorial&submenu=Chart_Data ). charts.swf seems a proper soft for producing charts, but have to work out the transformations :(

under project SmallUtils:
A simple SaxParser that converts the log files produced to csv, and appends all together in csvTestResults.csv.



Some ideas for next tests:
1. Measure broker disk consumption, and try to push it to 100% (maybe increase message payload size? )
2. test with many producers;
3. test with consumers.

for automation, we'll need to have scripts allowing to subscribe and unscribe (use wassh?)

and that's all folks!

Ok, finally ready to run the first test,

Using netcat, and under the approximate same conditions of woijtek.

I created a small, simple shell script to measure the throughput and cycle in between. I will deploy it also in lxb6118 to get comparative results.

here's the script :)

#!/bin/bash
# @author dfrodrig
# createFileXMessages.sh
# this program does the following:
# 1. creates an $expIteration number of messages on base 2, each message is $messageTemplate.
# 2. sends the created messages to $BROKER
# 3. outputs the time needed to send the n messages to $timeLogFile (Stomp n SEND command with one CONNECT and one DISCONNECT)

messageTemplate=messageTemplate
expIteration=21
timeLogFile=timedResults$expIteration.log
BROKER='lxb6117 6163'
testRunComments=$1

echo '' >> $timeLogFile
echo ''`date`''$HOST''$BROKER'' >> $timeLogFile
echo ''$expIteration'' >> $timeLogFile
echo '' `cat $messageTemplate` '' >> $timeLogFile
echo '' $testRunComments '' >> $timeLogFile

echo "#######################################################"
echo "# #"
echo "# GMAN tests for Amq #"
echo "# Sending 2^"$expIteration " messages to " i $BROKER
echo "# Results output to " $timeLogFile.
echo "# Messages template: " $messageTemplate
echo "# #"
echo "#######################################################"

#creates 2^numberMessages messages
for j in `seq $expIteration`
do
filename=Messages$j
echo 'Iteration ' $j ' Filename ' $filename
if [ -r $filename ]
then
rm $filename
fi
cat $messageTemplate > $filename
for i in `seq $j`
do
cat $filename $filename > MessagesTmp
mv MessagesTmp $filename
done
numberMessages=`grep -a SEND $filename | wc -l`
echo ''$numberMessages'' >> $timeLogFile
echo 'Timing.' $numberMessages ' messages sent from: ' $filename
#For some reason, the time being used by bash is not the /usr/bin/time, even though it seems to be a proper alias :S
#read: http://freebsd.amazingdev.com/blog/archives/000873.html for more info
echo '' >> $timeLogFile
/usr/bin/time -p --append --output=$timeLogFile cat prod $filename prodend | nc lxb6117 6163 > ncoutput
echo '' >> $timeLogFile
echo '' >> $timeLogFile
echo 'Iteration #' $j 'finished '
echo '#'
rm ncoutput
rm $filename
done

echo '' >> $timeLogFile
echo 'Program finished. Test Results output to ' $timeLogFile

I have started running some tests, I'll try to use wassh too. Now it's time to start thinking about which tests should i run.

very interesting...

or not so much:

> executing time is running something different from the expected from the command line :S

check out this discussion, seems to be the reason!
http://freebsd.amazingdev.com/blog/archives/000873.html

Meeting with Piotr

We were going through the use of netcat... much simpler than i thought, and now everything is a bit more clear :)

to begin with, something was not working, probably due to a problem in the Broker! (python consumer, as i can probably recall from previous posts

so:

CONNECT
SEND
SUBSCRIBE
DISCONNECT

are the commands STOMP that can be used, having the port 6163 open in the broker

other useful commands : grep -a SEND | wc -l : count the occurences of SEND in a particular line. ( useful for when creating lots of messages. )

So, the idea for the scripts was:
1. Create a consumer:
bash> nc lxb6117 6163 < cons > someLogFile

2. Create a producer :
bash> time cat prod msgd prodend | nc lxb6117 6163 # gets the time wasted to send all this messages.

After this tests, all the others where based on the python scripts. An additional time measurer was being done by the Java files : this would count messages and time, and publish to a different topic the performance.

and now, to work.


UPDATE: also useful is the stomp protocol information: http://stomp.codehaus.org/Protocol
Note: Ctrl+alt+2 for getting the null connection string :S

Got myself a name for the project!

GMAN!

Stands for GridMonitoringActiveNoting, or GridMonitoringActiveNoyan

Noyan:

Noticing:

GMAN can have 3 types of agents : Subscriber, Publisher, Broker.

Some exception types would be:
InvalidArgumentException
...

James log

1381 locate activemq.war
1382 cd
1383 ls
1384 cd apache-activemq-4.1.0-incubator
1385 l
1386 ls
1387 ls lib
1388 cd conf/
1389 ls
1390 less log4j.properties
1391 less activemq.xml
1392 ls
1393 ls -l
1394 less broker.ts
1395 cd ..
1396 ls
1397 find . -name \*.xml
1398 cd example/conf/
1399 ls
1400 less web.xml
1401 less activemq.xml
1402 less resin-web.xml
1403 cd ..
1404 ls
1405 ant
1406 ant war
1407 ls
1408 cd target/
1409 kl
1410 sls
1411 ls
1412 jar tf activemq-web
1413 jar tf activemq-web.war
1414 ls /var/log/
1415 ls
1416 history | more

Seems interesting enough to be used:

JMeter...

Apache JMeter is a 100% pure Java desktop application designed to load test functional behavior and measure performance. It was originally designed for testing Web Applications but has since expanded to other test functions.

http://jakarta.apache.org/jmeter/

I'll give it a shot this afternoon.

Inside GmsConsumer.xml

So, regarding lxb6117, we already concluded everything is set and working:

tomcat is containing a servlet which starts the AMQ service.
AMQService is configured as specified in the previous posts.
an mysql database and a journal are available. Data regarding messages is kept under /var/lib/tomcat5/activemq-data/localhost. I believe messages are therefore not stored in the database but in the journal. Only a reference to the journal is kept!

within lxb6118, for some reason, tomcat6 is being run instead. Most configuration lies in /opt/gms, as redirected from /etc/tomcat6/Catalina/localhost/GmsSameConsumer.xml

restarting tomcat service, allow GmsSameConsumer to be started.


Logging for GmsConsumer is located at:
tail -f /tmp/MessageConsumer-error.log
and
tail -f /tmp/MessageConsumer-debug.log


for lxb6117, no log is clearly defined (amq is probably using it's own logging mechanism? Seems probable!!)

configuracao no lbx6117

locais habituais do tomcat

E, em particular:
/opt/gms/AmqStarter!!!

Ok, as mensagens estao a passar do publisher e a chegar ao nosso amigo broker. A base de dados do activemq e preenchida, embora exista uma lag ( ler um bocado sobre o activemq torna-se imperativo!!

O admin para o activemq esta localizado no seguinte local: http://lxb6117.cern.ch:8161/, conforme explicitado em
http://note19.com/2007/06/23/configure-activemq-with-mysql/
Adenda: s'o a partir do activemq4.2!!



for monitoring activemq, before activemq 4.2, we should use JMX:

> jconsole &
http://activemq.apache.org/jmx.html
The remote url to be used: is "service:jmx:rmi:///jndi/rmi://lxb6117:1099/jmxrmi"

Work, and was almost forgetting to summarize... not so clever :(

So, today I installed eclipse, release Europa for j2ee developers under /opt/eclipse.

simply downloaded, and extracted.

>gunzip blahblah
>cd /opt
>tar-xvf blablah

before that, i lost a little bit of time in writing a small script to initialize a workspace env variable if i open a terminal from my machine: pcitgd24... plus, it creates a hardlink to where the code is located ( /home/dfrodrig/workspace ).

then came the cvs checkout. using kerberos,

> export CVSROOT=:kserver:glite.cvs.cern.ch:/cvs/glite #I should consider putting this in the .bashrc too :S

and finally:

> cvs co GmsSameConsumer

so, the code is in my hands. Also, we should notice that I haven't read report1.pdf and report2.pdf, but this is under the documentation and has the test results up to now! Plus, it informs how things had beeen done.

In the meantime, i recalled how to use a cluster of machines for sending messages: this should be wassh... maybe I should fetch the old code somewhere :S



The cvs configuration followed the details written here:
https://twiki.cern.ch/twiki/bin/view/EGEE/DevelopersGuide

although i use a kerberos to connect through the command line, eclipse only offers either pserver or extssh, so I'm using the latest.. let's see if eclipse does what it's supposed to!

Todays summary

after spending time setting up the pc ( still can't use dual head, will wait for the docking station ), and doing the security course + writing the french card, in the afternoon i got into the machines.

Current configuration, for testing tomorrow, is based on a tomcat broker service running on

http://lxb6117.cern.ch:8080/AmqStarter/

we start the broker by clicking on the button.

On lxb6118, the SAME consumer is on /opt/gms. there seems to be no tomcat running, still have to find how to start it and generate graphics and so on... also, there should be a producer somewhere, have to find it! Cross check with the cvs code, and should be easier to find.

Tomorrow, we work with a purpose!! from 9a.m, no distractions. and now, swim...

Tomorrow tasks

Morning:

beaureaucratic: Safety course 9h:French, 10h English.
Fill up the French Card form. take the necessary pictures.
Finish the laptop set-up. ( check what the heck is wrong with firefox. and include gnome blogger poster)
Think how to use the available machines in the most suitable way ( development on pcitgd10 + usual work on laptop? )

Afternoon:
Start testing lxb6117 and lxb6118
first changes to the code. compile where possible.

and there i go

I just cleanesed my previous ssh encarnation on afs...

bash-3.00$ cd public/
bash-3.00$ ls
authorized_keys id_dsa.pub identity.pub id_rsa.pub
bash-3.00$ ls -la
total 15
drwxr-xr-x 2 dfrodrig gm 2048 Sep 22 2006 .
drwxr-xr-x 39 dfrodrig gm 8192 Sep 19 16:24 ..
-rw------- 1 dfrodrig gm 336 May 30 2006 authorized_keys
-rw-r--r-- 1 dfrodrig gm 22 Oct 7 2004 .forward
-rw-r--r-- 1 dfrodrig gm 616 May 30 2006 id_dsa.pub
-rw-r--r-- 1 dfrodrig gm 345 May 30 2006 identity.pub
-rw-r--r-- 1 dfrodrig gm 236 May 30 2006 id_rsa.pub
bash-3.00$ rm *
bash-3.00$ cd ..
bash-3.00$ cd private/
bash-3.00$ ls
id_dsa identity id_rsa
bash-3.00$ ls -la
total 13
drwxr-xr-x 2 dfrodrig gm 2048 Sep 18 12:05 .
drwxr-xr-x 39 dfrodrig gm 8192 Sep 19 16:24 ..
-rw------- 1 dfrodrig gm 736 May 30 2006 id_dsa
-rw------- 1 dfrodrig gm 541 May 30 2006 identity
-rw------- 1 dfrodrig gm 951 May 30 2006 id_rsa
bash-3.00$ rm *
bash-3.00$ ls
bash-3.00$ cd ..
bash-3.00$ ls
bin Desktop dpm globus-old identity.pub mail private public scripts test.properties Thumbs.db webpage
bash-3.00$ cd .ssh/
bash-3.00$ ls

And generated new keys! All because of access to a ssh gateway to service:
https://twiki.cern.ch/twiki/bin/view/LCG/SSHGateway

voilá!

bash-3.00$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/afs/cern.ch/user/d/dfrodrig/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /afs/cern.ch/user/d/dfrodrig/.ssh/id_rsa.
Your public key has been saved in /afs/cern.ch/user/d/dfrodrig/.ssh/id_rsa.pub.
The key fingerprint is:
a2:dd:42:c9:17:8d:23:0d:76:44:16:15:92:a1:e4:54 dfrodrig@pcitgd10.cern.ch

and here's the public id_rsa, which was already sent for verification!

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2Y+bYbPZKhswRuJkgfQmV9Tfl02RaBue1KYqXS66rwiQSjSM4+kdSzsY9kfsmnIhJCItzdnRtvYdnfA7bV7Bh8IzxF/27AliKMOW35nYJohQL851B+amM9dpjtUmeRC5vladvfipdEo4q7FclsZ+H4gv6+AHfKgnHj0PCAQNkkNwFF3ksiKftgIjOp3ewIvSqIexvpsigPcJTKu5TVpdrgxDkFuRIiHRX0ka+SMXa956IuVvGzavt2CT870uzcIa3eu1nihkFr88OX6lgsf5qWCTZKWpD5gQz3k2A5+RpW4VoNyBAmEm4UcTwD+BWU1M9PLgxLXgWWxx4z8/ovu60w== dfrodrig@pcitgd10.cern.ch

fun, fun, fun!

Got my new laptop, a brand new HP nc6400.

After registering the new machine under the pcitgd24, I noticed that there are some problems! Here's the description:

HP Compaq NC 6400
* Please followup standard installation procedure.
* When system reboots on the Firstboot Display screen run, set monitor to Generic LCD Display -> LCD Panel 1280x800. (this can be also done later with system-config-display).
Update: as we have discovered only after CERN bought this hardware, the graphics chip is not fully supported by the Intel i810 driver. Instead, the vesa driver is being used, which cannot handle the exact display resolution offered by the hardware (1280x960 instead of 1280x800). You will need to apply additional workarounds to get the hardware resolution. We hope that a later driver update will fix this issue.
* After your system reboots, followup installation procedure for Intel Wireless Pro 3945 card driver.
* test results

ok, iĺl check this in a minute... still following the instructions ;)
http://linux.web.cern.ch/linux/scientific4/docs/install.shtml


UPDATE:
while having the fun, they suggest to change a lot in the "Monitor section" of /etc/X11/xorg.conf
this seems weird, so here's the old line's which have been removed!

HorizSync 31.5 - 90.0
VertRefresh 60.0 - 60.0

UPDATE2: screen resolution is now ok... still, "The above configuration allows to use the monitor's native resolution, but will still not give 3d acceleration or Xvideo overlay capabilities." going for solving the second issue, installation procedure for Intel wireless!

Use aspects and Spring

I should not forget to try to use aspects and/or Spring... let's see.

this morning I read a little bit more into SAM and reviewed JMS. Selectors are, in the end, something pretty basic, simply associated with the message. I'll see further when testing.

Afternoon:
- finish amq reading;
- get the machines access;
- run the current existing tests;

Meeting with James Casey and Piotr Nyczyk

We went through Wojciech Czech presentation on Grid Messaging System.

Main conclusions drawn:
I will be using AMQ messaging system, which implements jms and offers lots of other features such as persistance queuing of messages, possibility of redundacy through master=slave replication, etc... check his presentation (on the e-mail James sent) for more info.

Personal Notes: Check on latency issues (maybe have priority on the messages configurable on the system? Is this possible?

TODO (tomorrow): Analyze code, published : pay attention that 'src' folder doesn't have anything.
Read specification + SAM + Naigos?;
Check if access to boxes was granted;
Read something on JMS Selectors ( Get a good book from library)
Extend the tests already done and reported in the presentation.


mailOnThisSubject:
" boxes:
lxb6117, lxb6118

AMQ Broker:
OpenWire: lxb6117:6166
STOMP: lxb6117:6163

CVS:
http://glite.cvs.cern.ch/cgi-bin/glite.cgi/same/gms/

GMS Publisher script:
https://twiki.cern.ch/twiki/bin/view/LCG/GridPublisherSpecification "

1st Openlab Meeting

Meeting with openlab.

Presentation on 'Virtualization Benchmarks' (D. Raffu) and 'Review of recent processor evolutions and trends' (S. Jarp)

Responsible for openlab is Sverre Jarp.

ideas coming out of this: KVMware is not the best solution.

TODO: check what what is SMP (a big difference was happening, and i didn't know why :S)
DONE: http://en.wikipedia.org/wiki/Symmetric_multiprocessing

Pre-blogger journal

17-9-2007

PC install
we got a pc from upstairs.
DHCP was already configured.
AFS profile copied, but was not working, with errors
""
root@pcitgd10# /afs/usr/local/etc/ccdbuser dfrodrig
Fatal error, ccdbinfo returned:

SELECT
ORA-00942: table or view does not exist
""

after browsing,
http://linux.web.cern.ch/linux/redhat6/install/step_by_step.html
http://linux.web.cern.ch/linux/redhat6/laptop/#TOC19

concluded a home directory should be created:
http://linux.web.cern.ch/linux/redhat6/laptop/#TOC9

therefore:
root@pcitgd10# ccdbuser -home /home/dfrodrig dfrodrig
~
root@pcitgd10# /usr/sbin/pwconv
~
root@pcitgd10# passwd dfrodrig
Changing password for user dfrodrig.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
~
root@pcitgd10# ssh dfrodrig@pcitgd10
dfrodrig@pcitgd10's password:
Last login: Mon Sep 17 10:51:11 2007
/usr/bin/X11/xauth: creating new authority file /home/dfrodrig/.Xauthority

not sure yet if everything is working properly. ( when logging with my local account is afs certificate immeadiately available? )


18-9-2007

For having myself authenticated in lxplus ( no need to type the password over and over ), I need to have a kerberos initialization, issuing the command:
'kinit'

Moreover, there was a sync problem related to time unphasing. Localhost and server were not within the same timeframe therefore. NTPD was not installed, and this was done by using
ncm-ncd (Node Configuration Dispatcher of the Node Configuration Manager subsystem.) [ncm-ncd --list add ntpd as not activated]

After that, authentication worked. Home directory when logging now redirects to the afs home directory;

[unsolved] Still to solve, verify why the authentication doesn't work the other way around: when logged on lxplus, I can't login automatically to my local machine(pcitgd10).


Next tasks, get a Grid certificate, and access cvs repository.


Yum is the automatic updater in use for machine configuration. (http://linux.web.cern.ch/linux/scientific3/docs/softwaremgmtyum.shtml)


11:52 Have E-mail! Changes password, and issued a request for Cern certificate:

bash-3.00$ pwd
/afs/cern.ch/user/d/dfrodrig/private
bash-3.00$ openssl req -new -out myrequest.csr
Generating a 1024 bit RSA private key
...............++++++
................................++++++
writing new private key to 'privkey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:CH
State or Province Name (full name) [Berkshire]:Geneva
Locality Name (eg, city) [Newbury]:Meyrin
Organization Name (eg, company) [My Company Ltd]:CERN
Organizational Unit Name (eg, section) []:IT-GD
Common Name (eg, your name or your server's hostname) []:Daniel Rodrigues
Email Address []:daniel.rodrigues@cern.ch

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:Mother's Maiden Name
An optional company name []:
bash-3.00$ ls
id_dsa identity id_rsa myrequest.csr privkey.pem
bash-3.00$ vi myrequest.csr
bash-3.00$ vi myrequest.csr

bash-3.00$ pwd
/afs/cern.ch/user/d/dfrodrig/Desktop
bash-3.00$ ls ../private/
id_dsa identity id_rsa myrequest.csr privkey.pem
bash-3.00$ openssl pkcs12 -export -inkey ../private/privkey.pem -in newcert.cer -out myCertificate.pks
Enter pass phrase for ../private/privkey.pem: /* tainada Process caps */
Enter Export Password: /* Igreja Process caps */
Verifying - Enter Export Password: