Wifi download utility


released: 10.5.2020





Wifi_download for MB03+, ver. 0.3- readme 6.5.2020
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Authors: PC part (PHP scripts)- Lanex, ZX code- Hood (that's me:), Busy- Wifi Bios+ very useful routines
Acknowledgements: Lanex, LMN, Busy, Shrek, Peg7g, Nihirash, Martin Borik and others from Slack discussion.

I had many questions to many people, who helped me a lot. The program is dedicated to all of them. Thank you, chaps.


General info:
~~~~~~~~~~~~~
This utility allows you to download a single file from a server which can be run either on internet or locally on your hdd placed within the same home network as your MB03+ Wifi. The pogram is still 48K only. For the speed's sake, it uses uses the whole 48K memory. It can be quite easily transfered to another platforms which would require: a) the new platform has to have its own Wifi bios, b) BASIC file needs to be easily and slightly adjusted, c) DMA transfer need to be very easily turned off (if your system does not support it), d) file operations for your system need to be written.

If you use standart BASIC with 32 chars a line, use "ws030_32$". For BASIC with 64 chars a line, use more convenient "ws030_64$". The BASIC returns error codes see below. The package also contains file "wa030_bord" which additionally has blinking border. Violet= comunication with the wifi module, yellow= disk operations, black= main routine.

From my experiments, the Wifi works rather unstable, I mean the speed and transfer reliability. But thanks to parity check all data are transfered all right.



Controls:
~~~~~~~~~
For easy coding, I made all GUI (well GUI is too exaggerated, would abbreviate it to just UI, User Interface:) in BASIC. These are control keys:

- N: input file name to download
- U: input URL (either a name (eg. "hood.specc.cz" or number eg. "192.168.0.103"), for server requirements, read further
- Enter: start download
- T: test PHP script on the server. Lets you check your server, if it supports the transfer.
- S: save application. Useful for preserving your own unique URL, you do not have to type it all over again when application starts the next time
- X: parity check toggle ON/OFF. I strongly recommend leaving it ON. The transfer is not reliable.
- O: turn Wifi OFF, if you need
- P: enter your path to the file within a server or your disk. Read further.
- P: print turn on/off, during download. It enhances transfer speed a little bit.
- R: back to the root
- SPACE: download abort





What's new in ver. 0.3.
~~~~~~~~~~~~~~~~~~~~~~~~

- new: download possible from any path from server
- new: disk buffer increased to 32 KB-> resulting in higher and more stable transfer speed
- new: new wifi bios 1.2 added
- new: DMA ldir added for higher speed
- new: PRINT can be turned off during transfer
- code optimisation for higher speed (main code+ bios+ SP placed in upper memory)


to-do
~~~~~
- try to optimise speed of my code. To read data from Wifi module and clean them from "IPD" metadata right away- should help increase the speed also.


Program speed in ver 0.3
~~~~~~~~~~~~~~~~~~~~~~~~
Speed quite improved as compared to previous version 0.21 (the speed was 2,8 Kb/sec) up to 5 Kb/ sec. Mainly, speed remains rather constant even with long files as opposed to ver. 0.21. These are speeds in ver. 0.3., compare them with version 0.21 below:
- 7KB= 0:03 sec
- 91KB= 0:19 sec
- 592KB= 1:56 sec
- 987KB= 3:23 sec
- 1990KB= 7:09 sec



Path
~~~~
Now you can set your download path in BASIC. It can be either absolute path (eg. C:/yourdir/) or relative (eg. /yourdir/). Key R in BASIC returns you to the root directory. SAFETY WARNING!! Be carefull, the php scripts have access everywhere on your local disk. So it can download any file on your disk. Be aware of that fact. Eg. if your local server has public IP than your disk is accessible to anyone.


Print turn off
~~~~~~~~~~~~~~
Printing on screen is turned off by default during download which increases download speed a bit. Press P during download to toggle print ON/OFF.


Error codes in ver 0.3
~~~~~~~~~~~~~~~~~~~~~~
1=send_string_error
2=STATUS not returned
3=fail to send server connection
4=server not connected
5=send_string_error
6=send command error
7=200 OK not found
8=Content-Length not found
9=2x empty line not found
10=received content length not found
11=received data not found
12=start of data not found
13=":" not after IPD
14=file size not found
15=disk error
16=user abort (SPACE pressed)
17=file does not exist (PHP returned NE)
18=problem with blocks (PHP returned ER)
19=file does not exist
20=test PHP not sucessfull






Wifi_download for MB03+, ver. 0.21- readme 13.4.2020
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For Authors, acknowledgements and basic info, see info on ver. 0.1. below.


What's new in ver. 0.21.
~~~~~~~~~~~~~~~~~~~~~~~~

- bugfix: "look for string" routine- timeout was not running+ another timeout fix
- bugfix: timeout added to receive data routine, results in no communcation errors. Superb.
- bugfix: if data contained "IPD" string, they were cleaned badly
- new: 32 bit length for downlad file added
- new: download speed improvement- speed= cca 2,8 KB/sec including data parity check+ save on disk. (it is 3,2 KB/sec without parity check and 4,2 KB/sec without parity check& disk save*.(1,99 MB file transfered in 19 minutes and 45 secs= 1,68 KB/sec).
- new: press& hold space to cancel download
- new: file size& progress display (at least some basic info)
- new: if info.php returns NE (File not Exist), display it, in BASIC error nr.19
- new: test if server supports PHP*done
- new: new PHP script for download, it returns checksum byte as the last byte in the block. Big thank to its author, Lanex.
- new: parity check added to ensure 100% reliability of data (I hope so:)
- new: turn off Wifi option


Program speed
~~~~~~~~~~~~~
Version 0.21 gives much more better speed performance than ver. 0.1 but still it is far from the target. Hopefully some future improvements will come. Although, the bigger the file, the slower the transfer:(( For the moment I found out the reason is disk save, will try to find out more about it in the future. Also, download slows down towards the end of file. For normal ZX file length this is almost unrecognisable. Here are examples of speeds, including data parity check and disk save:
- 7KB= 0:03 sec
- 91KB= 0:32 sec
- 592KB= 4:05 sec
- 987KB= 7:49 sec
- 1990KB= 21:06 sec



PHP server test
~~~~~~~~~~~~~~~
Press "T" in BASIC to test if your server supports PHP properly. But first copy "test.php" file to the server. The server should return string "PHP OK" if everything is ok. If it does not return "PHP OK" the download will not work.


Data parity check
~~~~~~~~~~~~~~~~~
Can be turned off (toggled) in the BASIC program with "X" key. Although I do not recommend turning the feature off for reliability of downloaded data. Parity check also slows down downloaded data by cca 4 sec in 91 KB (91KB is my referential file where I measure speeds). And as you will see, parity errors, though rarely, will happen. Error counter is set to 255, if you exceed this number the program will end.



Colour version
~~~~~~~~~~~~~~
Find it in a package, same as normal one, but border in colour shows routine times. Magenta= handling http request+ downloading 2KB data, yellow= disk save, black= all the rest.






Wifi_download for MB03+, ver. 0.1- readme 2.4.2020
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Authors: PC part (PHP scripts)- Lanex, ZX code- Hood, Busy- Wifi Bios+ very useful routines
Acknowledgements: Lanex, LMN, Busy, Shrek, Peg7g, Nihirash


Intro
~~~~~
This very simple utility serves for files transfer between ZX and PC. Its main target is to avoid repeatedly removing memory card from MB03+ when copying files from/to it.


ver.0.1 features
~~~~~~~~~~~~~~~~
- downloads only one file, its name to be entered manually
- no upload
- can download both from local server or any internet server


server requirements
~~~~~~~~~~~~~~~~~~~
- local or any server needs to support PHP
- tested with Abyss local server (https://aprelium.com/abyssws/download.php) and ilnx.cz domain


how to install and run local server
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Local server enables you to copy files directly from you PC on your table. The main prerequisite is that your ZX and PC have to be connected within one network. But first you need to install it (see a tutorial: https://www.youtube.com/watch?v=YMYRLtfel38). Then you need to add PHP support to it. Installation of PHP support here: https://www.youtube.com/watch?v=zjYC4eRnRC0.

I am not going to give you step-by-step guide here. Just a principle. After you installed PHP, you need to set a path to your files in the Abyss server console. They are visible just like your any files in any file manager you use. Whatever you copy into this path, will be accessible on ZX by the wifi-download utility.

Set your host to run on port 80. If you have problems with running the host, try to copy "log" directory from Abyss Web Server folder to your local path/folder. It helped in my case. For sucess, the host on port 80 have to be running.

The last thing you need to do, is setup the local server for our purposes, ie. to copy all php files that you find in the utility package to your local folder. The utility needs them for its work.



Servers that are placed on the internet are quite easy to operate for utility purposes. If your server name is "testit.com", then type in simply "testit.com" in BASIC. Accessing local server, however, is a little different. Local server that you have just installed have to run on the PC that is connected to the same home or loacal network as your ZX. (Eg. there is a router in your house to which all Wifi (or LAN) devices are connected.

Every computer or device (like mobile phone, TV, electric tin opener:), etc.) has its own IP address. And the Wifi utility running on ZX have to connect just to this address. Your PC will tell you the address. If you are a Windows user open the command line and type in "ipconfig". In the output list you have to find your PC IP address which looks like "192.168.0.103". And you type exactly this format into BASIC. A hint: IP address can change. In order not to change it every time you power on your PC, go to your router settings and bind your PC with firm IP address, using PC's unique MAC number.



known problems, errors
~~~~~~~~~~~~~~~~~~~~~~
- the transfer is quite slow (actually, annoying slow)
- your wifi module have to be connected to AP (access point)
- on the output, if "error=0", all is ok. Any other number means error as follows:
1=send_string_error
2=STATUS not returned
3=fail to send server connection
4=server not connected
5=send_string_error
6=send command error
7=200 OK not found
8=Content-Length not found
9=2x empty line not found
10=received content length not found
11=received data not found
12=start of data not found
13=":" not after IPD
14=file size not found
- sometimes, initial connection to the server is not sucesfull and returns error, just try again, I need to work on it yet.
- length of file is max 65535 (will be soon extended to 24 or 32 bit length)
- if special characters in file name (like e.g. "+"), download will not work


other thoughts
~~~~~~~~~~~~~~
- version 0.1 is the least user friendly version, it uses BASIC to enable user change name of the desired file and URL
- use S in BASIC to save your local server URL for downloading next time
- I know, there is a lot of setting up before you can start your work. On ZX side some more user friendly client would be needed. Maybe in the future.


how it works
~~~~~~~~~~~~
ZX sends HTTP request to the PC server and its PHP scripts return answers consisting of http headers and bodies. First, a PHP script returns file length and in second step another PHP script returns file content cut down to 2KB parts (exactly the same length as incoming buffer in our ESP-01 wifi module). So there is no loss of bytes. After ZX has saved the 2KB data, ZX asks for another 2KB part. Unfortunatelly, for every 2KB part a single http request have to be made, which is one of the reasons for utility slowness. In the upper third of ZX screen you can see what is being sent to a server. It all makes one http request.





*****************************************************************************************

Remarks for me:
~~~~~~~~~~~~~~~
časy:

buffer delky #6000, kod i buffer nad #8000:
stah= 23 s 3,95 kb/s
vy= 2:34s 3,84 kb/s
go= 4:20 3,79
pl= 9:08 3,63

buffer #6800, kod i buffer nad #8000
vy= 2:31

buffer=#8000. start bufferu #63e0, kod nahore
stah= 23 s 3,95 kb/s
vy= 2:30s 3,94 kb/s
go= 4:14 3,89
pl= 8:50 3,75


special kod 5217 bajtu
vy= 2:12

kod 5229 bajtu
vy= 2:16

buffer=#8000, start bufferu #63e0. kod 5229 bajtu nahore, vypnute CLS i PRINT nahore, zasobnik pod #63e0
stah= 19 s 4,78 kb/s
vy= 2:04s 4,77 kb/s

buffer=#8000, start bufferu #63e0. kod nahore, zasobnik nahore, moc rychle, nedocitava data

buffer=#8000, start bufferu #63e0. kod nahore, zasobnik dole
vy= 2:08 ldir DMA
vy= 2:15 klasicky ldir

buffer=#8000, start bufferu #63e0. kod nahore, zasobnik dole, ldir dma, vypnute print a cls
vy= 1:58 5,01 kb/s (3x chyba parity)
1:55 5,14 kb/s (0x chyb parity)-- rezerva je jeste v uisteni zasobniku v rychle RAM, ale modul nedocita bajty a musi se zvetsit timeout cteci rutiny, cimz se to cele zase zpomaluje
stah= 0:18 5:05 kb/s

1xerr,old,2:08 delka souboru 5332
new 2:05 delka souboru 5397
2:03 bez clean line, delka 5366, nove 5339

pl 7:09 4,63 KB/sec
go 3:23 4,86 kb/sec
stah 0:19 4,78 kb/sec
vy 1:56 5,10 kb/sec s novym biosem 1.2, SP=0, 5,10 Kb/sec
kdyz vypnu ukladani na disk, pak je cas 1:45= 5,63 Kb/sec
bez disku a k tomu cisteni dat


- Udelal jsem test na cisty prenos a ten skutecne je pri nastaveni seriaku 115200 b cca 11KB/sec. Kdyz jsem na to pustil i rutinu na procisteni dat od IPD..., klesla rychlost na 6,4 KB/sec.
- zvysit prenos z bloku 2KB na 16 nebo 32, ale zde vzdy modul posle o 1,5 kb dat mene nez chci. A standartne je uzavira retezcem CLOSED.