Software Design by Example: Transferring Files

<p>
  <a href="https://third-bit.com/sdxpy/">Software Design by Example</a>
</p>
<h1>Transferring Files</h1>
<div class="bottom">
  <a href="../">chapter</a>
</div>

---

## The Problem

-   We want to make information available to others

-   Which includes programs as well as people

-   Understanding how a web server works will help us do this

---

## TCP/IP

-   Most networked computers use <a class="gl-ref" href="../../glossary/#gl:internet_protocol" title="A set of specifications for ways computers can communicate. TCP/IP is the most widely used." markdown="1">Internet Protocol (IP)</a>

-   Defines multiple layers on top of each other

-   <a class="gl-ref" href="../../glossary/#gl:tcp" title="The most popular member of the IP family of protocols. TCP/IP tries to deliver messages reliably and in order so that programs can communicate as if they were reading and writing files." markdown="1">Transmission Control Protocol (TCP/IP)</a>
    makes communication between computers look like
    reading and writing files

---

## Sockets

-   A <a class="gl-ref" href="../../glossary/#gl:socket" title="A communication channel between two computers that provides an interface similar to reading and writing files." markdown="1">sockets</a> is one end of a point-to-point communication channel

-   <a class="gl-ref" href="../../glossary/#gl:ip_address" title="A four-part number that uniquely identifies a computer on a network." markdown="1">IP address</a> identifies machine

-   Typically written as four 8-bit numbers like `93.184.216.34`

-   <a class="gl-ref" href="../../glossary/#gl:port" title="A logical endpoint for communication, like a phone number in an office building. Only one program on a computer may use a particular port on that computer at any time." markdown="1">port</a> identifies a specific connection on that machine

-   A number in the range 0–65535

-   Some numbers reserved for well-known applications

-   E.g., port 80 is usually a web server

---

## Naming Things

-   IP addresses are hard to remember

-   And might actually identify a set of machines

-   <a class="gl-ref" href="../../glossary/#gl:dns" title="A decentralized naming system for computers that translates hostnames into the IP address of particular computers." markdown="1">Domain Name System (DNS)</a> translates names like `third-bit.com`
    into numerical identifiers

---

## Clients and Servers

-   A <a class="gl-ref" href="../../glossary/#gl:client" title="A program such as a browser that sends requests to a server and does something with the response." markdown="1">client</a> sends requests and processes responses
    (e.g., web browser)

-   A <a class="gl-ref" href="../../glossary/#gl:server" title="A program that waits for requests from clients and sends them data in response." markdown="1">server</a> waits for requests and replies to them
    (e.g., a web server)

---

## Socket Client

```py
import socket

CHUNK_SIZE = 1024
SERVER_ADDRESS = ("localhost", 8080)

message = "message text"

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(SERVER_ADDRESS)
sock.sendall(bytes(message, "utf-8"))
print(f"client sent {len(message)} bytes")

received = sock.recv(CHUNK_SIZE)
received_str = str(received, "utf-8")
print(f"client received {len(received)} bytes: '{received_str}'")
```

---

## Socket Server

```py
import socket

CHUNK_SIZE = 1024

def handler():
    host, port = socket.gethostbyname("localhost"), 8080
    server_socket = socket.socket()
    server_socket.bind((host, port))

server_socket.listen(1)
    conn, address = server_socket.accept()
    print(f"Connection from {address}")

data = str(conn.recv(CHUNK_SIZE), "utf-8")
    msg = f"got request from {address}: {len(data)}"
    print(msg)

conn.send(bytes(msg, "utf-8"))
    conn.close()

```

---

## Interactions

---

## Using the Library

```py
import socketserver

CHUNK_SIZE = 1024
SERVER_ADDRESS = ("localhost", 8080)

class MyHandler(socketserver.BaseRequestHandler):
    def handle(self):
        data = self.request.recv(CHUNK_SIZE)
        cli = self.client_address[0]
        msg = f"got request from {cli}: {len(data)}"
        print(msg)
        self.request.sendall(bytes(msg, "utf-8"))

if __name__ == "__main__":
    server = socketserver.TCPServer(SERVER_ADDRESS, MyHandler)
    server.serve_forever()
```

---

## Chunking

-   Server uses `self.request.recv(CHUNK_SIZE)`

-   What if client sends more data than that?

-   Allocating a larger <a class="gl-ref" href="../../glossary/#gl:buffer_memory" title="A temporary storage area in memory." markdown="1">buffer</a> just delays the problem

-   Better solution: keep reading until there is no more data

---

## Reading Chunks

```py
class FileHandler(socketserver.BaseRequestHandler):
    def handle(self):
        print("server about to start receiving")
        data = bytes()
        while True:
            latest = self.request.recv(CHUNK_SIZE)
            print(f"...server received {len(latest)} bytes")
            data += latest
            if len(latest) < CHUNK_SIZE:
                print(f"...server breaking")
                break
        print(f"server finished received, about to reply")
        self.request.sendall(bytes(f"{len(data)}", "utf-8"))
```

---

## Writing Chunks

```py
def send_file(conn, filename):
    with open(filename, "rb") as reader:
        data = reader.read()
    print(f"client sending {len(data)} bytes")
    total = 0
    while total < len(data):
        sent = conn.send(data[total:])
        print(f"...client sent {sent} bytes")
        if sent == 0:
            break
        total += sent
        print(f"...client total now {total} bytes")
    return total
```

-   Try to send all remaining data

-   Advance marker by amount actually sent and re-try

---

## Output

-   Client

```
client sending 1236 bytes
...client sent 1236 bytes
...client total now 1236 bytes
client main sent 1236 bytes
client main received 1236 bytes
True
```

- Server

```
server about to start receiving
...server received 1024 bytes
...server received 212 bytes
...server breaking
server finished received, about to reply
```

---

## Testing

-   Manual testing:
    -   Start the server
    -   Wait for it to be ready
    -   Run the client
    -   Shut down the server

-   Better: use a <a class="gl-ref" href="../../glossary/#gl:mock_object" title="A simplified replacement for part of a program whose behavior is easy to control and predict. Mock objects are used in unit tests to simulate databases, web services, and other complex systems." markdown="1">mock object</a>
    instead of a real network connection

---

## Refactor the Logging

```py
class LoggingHandler(socketserver.BaseRequestHandler):
    def handle(self):
        self.debug("server about to start receiving")
        data = bytes()
        while True:
            latest = self.request.recv(BLOCK_SIZE)
            self.debug(f"...server received {len(latest)} bytes")
            data += latest
            if len(latest) < BLOCK_SIZE:
                self.debug(f"...server breaking")
                break
        self.debug(f"server finished received, about to reply")
        self.request.sendall(bytes(f"{len(data)}", "utf-8"))

```

```py
def debug(self, *args):
    print(*args)
```

---

## Creating a Testable Server

```py
class MockHandler(LoggingHandler):
    def __init__(self, message):
        self.request = MockRequest(message)

def debug(self, *args):
        pass
```

-   *Don't* upcall constructor of `LoggingHandler`

-   Don't want to trigger all of the library's socket machinery

---

## Mocking the Request Object

1.  A constructor that records
    the data we're going to pretend to have received over a socket
    and does other setup

2.  A `recv` method with the same signature as the real object's

3.  A `sendall` method whose signature matches the real thing's

---

## Mocking the Request Object

```py
class MockRequest:
    def __init__(self, message):
        self._message = message
        self._position = 0
        self._sent = []

def recv(self, max_bytes):
        assert self._position <= len(self._message)
        top = min(len(self._message), self._position + BLOCK_SIZE)
        result = self._message[self._position:top]
        self._position = top
        return result

def sendall(self, outgoing):
        self._sent.append(outgoing)
```

---

## Our First Test

```py
def test_short():
    msg = bytes("message", "utf-8")
    handler = MockHandler(msg)
    handler.handle()
    assert handler.request._sent == [bytes(f"{len(msg)}", "utf-8")]
```

-   Trade <a class="gl-ref" href="../../glossary/#gl:test_fidelity" title="The degree to which a mock object or other replacement for part or all of a system mimics the behavior of that system for testing purposes." markdown="1">fidelity</a> for ease of use

---

## Summary