■ Exercise 4
2. Addition of CSV data lines to HTML data on web control screen
3. About the installation of each command
4. Example of acquisition of pages, filtering and editing
5. How to schedule regular command execution
6. Upload of CSV file to ftp server
→Click here if you have some questions and comments.
←Back to the table of contents
Updated May 30, 2005 18:33
●Goal of Exercise 4
As Exercise 4, submit the exercise of "Acquisition of control page including CSV format data and upload to FTP server". →Click here for the submission of exercise.
Utilize actually some of items you mastered in IT graduate school curriculum: (A) Acquisition of a minimum of knowledge and skills to become an IT engineer.
※Relationship diagram of order of courses：
http://cai.cs.shinshu-u.ac.jp/sugsi/info/LecOrdIT.png
There are two objectives in this exercise.
(1) Creation of CSV format data (PICNIC side)
Based on the door security system (ver1.2.0.8) created in Exsercise 3, add CSV (comma-separated format) lines including various data which begin with "CSV," to HTML data on web control screen.
(2) Upload of acquisition data to FTP server (PC side)
By using the following acquisition tool of HTML pages and various editor with Unix (such as Linux, FreeBSD and Sun) or WindowsPC which can be connected to PICNIC, acquire pages, detect, filter and edit CSV lines and finally upload CSV data edited to FTP server.
・wget: acquisition program of HTML pages
・grep：filter
・sed ：stream editor
・date：acquisition and display of date/time
Because of highly versatile data format, it can be used for many purposes such as database as logs and understanding of recent state of each PICNIC node.
＜Subjoinder＞
How to use each command is displayed by "man electronic-manual-page" or
adding the "--help" option (e.g.: wget --help).
●Addition of CSV data lines to HTML data on web control screen
To make editing work after acquisition of HTML data easy, add various data lines which begin with "CSV," to HTML data on web control screen. It's desirable to add after beginning a new line (\n) in the last line of the footer.
(1) CSV data format
Sequence of comma-separated data of CSV lines is as follows:
※This time, IP address and port number are not listed on the data.
In some applications, they should be listed.
＜CSV data format (actually one line)＞
-------------------------------------------
CSV,
Student ID number,
Firm Version,
MAC Address,
RA0,RA1,RA2,RA3,RA5,
RB0,RB1,RB2,RB3,RB4,RB5,RB6,RB7,
Counter for the number of times of opening and closing the door (end of line)
-------------------------------------------
＜Output example＞
CSV,02TA699X, 1.2.0.9, 00:02:CB:01:2E:75, 0, 0, 0, 0, 27, High , Low , High , High , High , Low , Low , Low , 0
→Click here to see the actual web page (HTML).
(2) Change of program
Add the following codes to the footer of message bank #4 on the web control screen.
＜Additional codes to the footer of message bank #4 on the web control screen＞
; |
●About the installation of each command
(1) In the case of Unix machine
Shells such as sh and bash, various commands such as grep, sed and date and wget are already installed as a standard installation in recent Linux-OS and so on.
(2) In the case of WindowsPC
There are the following two methods.
[A] Installation of Cygwin environment.
[B] Installation of GNU Unix compatible utility which runs in the Win32 environment.
[A] Cygwin environment
Execute Cygwin environment setup from the Cygwin homepage.
http://www.cygwin.com/
http://www.cygwin.com/setup.exe
grep, sed, date: default installation of the Base category
wget: choose the installation of wget in the Web category (cannot be installed by default).
These are operated on the bash command shell.
[B] Win32-Unix compatible command environment (2005.3.25 revised again)
Install Unix compatible commands with GNU utilities for Win32.
After downloading from this local copy (12.6MB), execute unxutils.exe to setup.
Choose Basic at the time of installation.
Command group: installed under C:\Windows\System32\ (therefore, it is not necessary to change the command path.)
Manual: installed under C:\Program Files\UnxUtils\
Each compatible command is used on the sh shell called from the DOS command prompt.
For how to use commands, refer to the electronic manual by man wget and so on.
In the case of WinXP:
【Start】→【All Programs】→【Accessories】→＜Command Prompt＞
Or,
【Start】→【Run】→ cmd.exe
Then, the sh shell compatible environment will be available by executing c:\windows\system32\sh.
It allows the same regular expression and redirection as shell commands on Unix (Cygwin) (extremely convenient!).
※Note In WinXp and so on, execute setup by administrator user whose name is composed of alphanumeric characters and who has administrative right.
●Example of acquisition of pages, filtering and editing (2005.3.25 revised)
※Hereinafter, it assume that PICNIC can be connected when IP address is 192.168.1.200 and http port number is 8080. (Leave the power on at the time of experiment.)
(1) Acquisition of pages of PICNIC by wget (Issuance of GET command)
Example of acquisition of HTML page:
wget http://192.168.1.200:8080/
→ Stored in index.html
wget -O picnic.html http://192.168.1.200:8080/
→ Stored in picnic.html
wget -O - http://192.168.1.200:8080/
→ Written out to standard output
wget -O picnic.html http://192.168.1.200:8080/modify.cgi?RB6=H
→ HTML response is stored in picnic.html after changing the RB6 port of PICNIC to High by GET method. (The wget command allows remote control of the port without a browser!) ※2005.3.25 revised (added the -O argument)
(2) Detection of CSV lines by grep
Pick up any lines including the character "CSV".
grep "CSV" inputfile.txt
(3) Addition of date/time information by sed
Save a regular expression of adding the output of date with a comma at the head ofline as date.sed.
echo "s/^/`date`,/g" > date.sed
※Note: In some Japanized Unix (such as Japanese rvt terminal of TurboLinux 8 workstation) environments, the output of date may be Japanese language because LOCALE is ja_JP.eucJP and so on. If you want to output date in English in an environment like this, execute date as described above after specifying LANG=C using the env command.
Eg.: env LANG=C date
＜Reference URL＞
Locale configuration
Next, add date/time at the top of the input file using this date.sed.
sed -f date.sed inputfile.csv
(4) Eventual script
Eg.: When adding to 02TA699X.csv
[A] In the case of Unix (Cygwin) environment
picnic-get.sh
-----------------------------------------------------------------------
echo "s/^/`date`,/g" > date.sed ; wget -O - http://192.168.1.200:8080/
| grep --text "CSV" | sed -f date.sed >> 02TA699X.csv
-----------------------------------------------------------------------
[B] In the case of Win32 environment
Call commands sequentially by sh -c as described below.
picnic-get.cmd
-----------------------------------------------------------------------
echo off
sh -c "echo s/^/`date`,/g > date.sed"
sh -c "wget -O - http://192.168.1.200:8080/ | grep --text 'CSV' >
picnic.csv"
sh -c "cat picnic.csv | sed -f date.sed >> 02TA699X.csv"
-----------------------------------------------------------------------
＜Example of the CSV file created＞
●How to schedule regular command execution
(1) In the case of Unix machine
Execute regularly shell script using the cron or at command.
(2) In the case of Cygwin environment and Win32 environment (OS-dependency)
[A] When the schedule service is running on Win2000, XP and so on:
The AT command allows executing tasks on schedule.
In the case of WinXP (classic menu):
Configure in【Start】→【Settings】→【Control Panel】→【Scheduled Tasks】.
※Only the minimum day-to-day schedule specifying the time can be configured by the task wizard.
[B] When using a free software which can schedule execution of commands:
eg.: Multi Function Alarm Ver.1.52 (free software)
http://member.nifty.ne.jp/T_sugiyama/#MFA
[C] Configuration to use cron on Cygwin:
Cron can be also used on Cygwin by installing cygrunsrv to NT service.
http://www.amy.hi-ho.ne.jp/tachibana/cygwin/cron.html
●Upload of CSV file to ftp server
Eg.: When the student ID number is 02TA699X
A CSV file named 02TA699X.csv exists in the current directory. Upload it by ftp to the incoming/picnic directory of the server using anonymous user.
There are the following two methods to upload automatically from an FTP client.
(1) Using ftp command input file (available in ftp client on the Cygwin environment where the -s option can be used)
(2) Preparing a file in which FTP commands are enumerated to redirect (available in ftp client on Unix where the -n option can be used)
(1) Using FTP command input file (available in ftp client on the Cygwin environment)
＜FTP command input file, ftplogin.txt＞
open ysserve.cs.shinshu-u.ac.jp |
＜Execution example＞ Execute "ftp -s:ftplogin.txt"
THINKPAD2# ftp -s:ftplogin.txt |
(2) Preparing a file in which FTP commands are enumerated to redirect (using the -n option)
＜File in which FTP commands are enumerated, ftpupload.txt＞
open ysserve.cs.shinshu-u.ac.jp |
＜Execution example＞ Execute "ftp -v -n < ftpupload.txt"
THINKPAD2# ftp -v -n < ftpupload.txt |
Access the following URL with your browser to confirm the upload.
ftp://ysserve.cs.shinshu-u.ac.jp/incoming/picnic/
CSV files stored in the above URL can be opened from InternetExplorer on a Windows machine where MS-Excel is installed.
＃ Do not delete other students' files!
●Submission of Exercise 4
|
【Exercise 4】(Integral experiment) 【Ａ】Creation of CSV format data (PICNIC side) 【Ｂ】Upload of acquisition data to FTP server (PC side) |
【Ａ】Creation of CSV format data (PICNIC side) and report submission
(1) Update firmware to display CSV format data on the web control screen.
(2) Using the state of door sensor, the changes of counter values and the web control screen, confirm working as required.
(3) Change the version number of the assembled source file to
ver1.2.0.9
And change arbitrarily the footer of the web control screen to your student ID number and so on.
(4) Save the assembled source file under the name of v1209e4.asm. (English one byte characters/ Lower-case alphabets/ Do not use one-byte katakana and double-byte characters as file name.)
(5) From the following report submission system, upload v1209e4.asm. We will mark your report after receiving it.
＜How to submit＞
For the program you created, submit the assembled source file (the extension is not *.hex but *.asm) by using the following report system (with the file upload function).
We will judge resubmission or completion after checking the operation of the file you submitted. You have to resubmit if there are any defects in the operation. You can "finish【A】" if the operation is OK.
Click here to submit the report of Exercise 4【A】(Exercise number: picnic-exp04).
(6) You have to "wait our judgement". Send an e-mail to here if you do not receive the result of judgement for a long time.
(7) If you received the judgement of resubmission, correct the defects indicated, change the filename to the name of v1209e4a.asm and so on and submit it.
(8) Without (7) (that is, the judgement of completion), your report Exercise 4【A】was accepted once. (But you have to go on Exercise 4【B】.)
【Ｂ】Upload of acquisition data to FTP server (PC side)
(9) Set shells such as sh and bash, various commands such as grep, sed and date, and wget as enabled with a PC which can be directly connected to PICNIC.
(10) After implementing the shell script of acquisition of pages, filtering and editing according to individual OS environment, confirm acquisition of the output of CSV implemented in the above【Ａ】from PICNIC, addition and storage of CSV file. For date and time, acquire 3 to 4 times at intervals of about ten seconds.(11) You don't have to configure the schedule of regular command execution at this moment. (Take your time to try at a later date.)
(12) A CSV file named student-ID-number.csv should be created in the current directory. Upload it by ftp to the incoming/picnic directory of the ftp server (ysserve.cs.shinshu-u.ac.jp) using anonymous user.
(13) Access the following URL to confirm that your CSV file was uploaded.
ftp://ysserve.cs.shinshu-u.ac.jp/incoming/picnic/
(14) After confirming uploading, input a comment including the URL of CSV file you uploaded to teacher comment field
＜How to submit＞
We will judge resubmission or completion after checking the URL of CSV file described in the comment field. You have to resubmit if there are any defects. You can finish【Ｂ】and the whole Exercise 4 in the case of OK.
Click here to submit the report of Exercise 4【B】(Exercise number: picnic-exp04).
(15) You have to "wait our judgement". Send an e-mail to here if you do not receive the result of judgement for a long time.
(16) If you received the judgement of resubmission, correct the defects indicated and resubmit it.
(17) Without (16) (that is, the judgement of completion), your report Exercise 4【B】and the whole Exercise 4 was accepted.
Students who finished the above (17), good work! That's the end of required exercises of "IT Technology exercise". We hope you will study and research aggressively in Shinshu University Graduate School on the Internet, join in the remote education project. Thank you for your cooperation. m(_o_)m
●Point of consideration
Point of consideration on this time's specification and implementation is as follows. You do not have to answer it as a report, but make sure to arrange your thoughts before you finish the program and submit your report.
[1] About this time's CSV output format, the version of firmware of PICNIC to be acquired and the data processing (which column means what) after acquisition have to be mutually collaborated and corresponded. On the other hand, from the data collection PC side, the "node" for getting HTML does not have to be an actually "existing" PICNIC microcomputer board. What this means is that there is a possibility for the number of I/O ports to change as a result of a version change in the PICNIC microcomputer board and that could change the assignments of each column. In the CSV data format used here, we simply put data in rows and do not have a title line for expressing the port assignment or meaning of each column. Therefore, it would be a good idea in the finalized *.csv file to include in the first line the appropriate "CSV file format version" (not the PICNIC firmware version) and the assignments for each column. Discuss the important points of consideration and implementation for this approach.
[2]
For the data collection from
the PICNIC->server upload, what would
be the appropriate collection period and
timing? This will depend greatly on
the application to be used. For example,
if you are going to use it for a
"temperature recording device",
it will probably be sufficient
to take measurements every 10
minutes -> upload a few hours worth
all at once. On the other hand,
if you are going to implement a
door security system and speed is
of the importance in alerting users
of door alarms during out modes, there
will probably be cases where
measurement -> alarm responses should
be less than one minute.
Consider
carefully the timing for data
measurements and delivery for
each application.
[3] By setting CGI parameters (such as ?RB6=H) with the wget command, the PICNIC port output control can be done instantly from a shell script. Consider some application examples that can use this feature.
●Reference URL
UNIX Tools for Win32
http://crocus.sci.kumamoto-u.ac.jp/physics/astro/fujimoto/home/junixwin32.htm
Cygwin
http://www.cygwin.com/
GNU utilities for Win32
http://unxutils.sourceforge.net/
GNU FSF/UNESCO Free Software Directory
http://www.gnu.org/directory/
wasaki@cs.shinshu-u.ac.jp
Copyright(c) 2005 Katsumi Wasaki. All rights reserved.