Custom Protocol Implementation Advanced

Thiết kế binary protocol cho high-performance systems

1. Tại sao cần Custom Protocol?

Protocol	Overhead	Parsing Cost	Use Case
HTTP/1.1	~500 bytes	String parsing	Web APIs
JSON	~40% overhead	UTF-8 decode	Human-readable
Custom Binary	0 overhead	Pointer cast	HFT, Games

---

2. Message Framing Strategies

Vấn đề: TCP là Stream

TCP không có "message boundaries" — dữ liệu đến như stream liên tục.

Strategy 1: Length-Prefixed Framing

┌──────────────┬──────────────────────────────────────┐
│   Length     │              Payload                 │
│   (4 bytes)  │          (N bytes)                   │
└──────────────┴──────────────────────────────────────┘

use std::io::{self, Read, Write};
use std::net::TcpStream;

pub struct LengthPrefixedCodec;

impl LengthPrefixedCodec {
    pub fn write(stream: &mut TcpStream, data: &[u8]) -> io::Result<()> {
        let len = data.len() as u32;
        stream.write_all(&len.to_be_bytes())?;
        stream.write_all(data)?;
        stream.flush()
    }
    
    pub fn read(stream: &mut TcpStream) -> io::Result<Vec<u8>> {
        let mut len_buf = [0u8; 4];
        stream.read_exact(&mut len_buf)?;
        let len = u32::from_be_bytes(len_buf) as usize;
        
        const MAX_SIZE: usize = 16 * 1024 * 1024;
        if len > MAX_SIZE {
            return Err(io::Error::new(io::ErrorKind::InvalidData, "Too large"));
        }
        
        let mut payload = vec![0u8; len];
        stream.read_exact(&mut payload)?;
        Ok(payload)
    }
}

Strategy 2: Delimiter-Based

// JSON Lines: mỗi message kết thúc bằng \n
// {"id":1}\n{"id":2}\n

Strategy 3: Fixed-Size Messages

#[repr(C, packed)]
pub struct OrderMessage {
    pub msg_type: u8,
    pub order_id: u64,
    pub price: f32,
    pub quantity: u32,
}

impl OrderMessage {
    pub const SIZE: usize = 17;
}

---

3. Endianness Handling

Big Endian (Network):    0x12 0x34 0x56 0x78  (MSB first)
Little Endian (x86):     0x78 0x56 0x34 0x12  (LSB first)

fn endianness_demo() {
    let value: u32 = 0x12345678;
    
    // To Big Endian (network byte order)
    let big = value.to_be_bytes();  // [0x12, 0x34, 0x56, 0x78]
    
    // From Big Endian
    let from_big = u32::from_be_bytes([0x12, 0x34, 0x56, 0x78]);
}

---

4. Serialization Framework Comparison

Protocol Buffers

// Với prost crate
#[derive(prost::Message)]
pub struct Order {
    #[prost(uint64, tag = 1)]
    pub order_id: u64,
    #[prost(string, tag = 2)]
    pub symbol: String,
}

// Serialize/Deserialize
let mut buf = Vec::new();
order.encode(&mut buf).unwrap();
let decoded = Order::decode(&buf[..]).unwrap();

Cap'n Proto (Zero-Copy)

// Đọc trực tiếp từ bytes - KHÔNG copy
let message = capnp::serialize::read_message(data, opts).unwrap();
let order = message.get_root::<order::Reader>().unwrap();
println!("{}", order.get_order_id()); // Zero-copy access

FlatBuffers (Zero-Copy)

// Zero-copy reading
let order = flatbuffers::root::<Order>(data).unwrap();
println!("{}", order.order_id()); // Direct memory access

Performance Comparison

Format	Wire Size	Parse Time	Zero-Copy
JSON	75 bytes	1500 ns	❌
Protobuf	28 bytes	250 ns	❌
FlatBuffers	48 bytes	15 ns	✅
Cap'n Proto	56 bytes	10 ns	✅
Raw struct	32 bytes	0 ns	✅

---

5. Zero-Copy Deep Dive

Traditional:  Network Buffer → Copy → Application Object
Zero-Copy:    Network Buffer ← Pointer ← View (no copy)

pub struct OrderView<'a> {
    buffer: &'a [u8],
}

impl<'a> OrderView<'a> {
    pub fn order_id(&self) -> u64 {
        // Pointer arithmetic only, no copy
        u64::from_le_bytes(self.buffer[0..8].try_into().unwrap())
    }
    
    pub fn symbol(&self) -> &'a str {
        // Returns slice into original buffer
        std::str::from_utf8(&self.buffer[8..16]).unwrap()
    }
}

---

6. HFT Protocol Example

const MSG_NEW_ORDER: u8 = 1;
const MSG_EXECUTION: u8 = 3;

#[repr(C, packed)]
pub struct MessageHeader {
    pub msg_type: u8,
    pub version: u8,
    pub length: u32,  // Big Endian
}

#[repr(C, packed)]
pub struct NewOrder {
    pub order_id: u64,
    pub symbol_id: u32,
    pub price: i64,      // Fixed-point: price * 10^8
    pub quantity: u32,
    pub side: u8,
}

impl NewOrder {
    pub fn price_as_f64(&self) -> f64 {
        i64::from_be(self.price) as f64 / 100_000_000.0
    }
}

---

🎯 Best Practices

Scenario	Recommendation
Microservices	gRPC + Protocol Buffers
Game networking	FlatBuffers
HFT systems	Raw structs + custom framing
Balanced	Cap'n Proto

💡 PROTOCOL DESIGN

1. Version field — Forward compatibility
2. Message type — First byte để route nhanh
3. Length prefix — Biết khi nào message complete
4. Big Endian — Network byte order convention