ESP32 HTTP/2: Decompressing Brotli encoded content

综合技术 2018-12-08 阅读原文

In this tutorial we are going to check how to do a HTTP/2 GET request, accepting the content compressed with the Brotli algorithm. Thus, we will later decompress it back, after receiving the response. The tests were performed using a DFRobot’s ESP32 module integrated in a  ESP32 development board .

Introduction

In this tutorial we are going to check how to do a HTTP/2 GET request, accepting the content compressed with the Brotli algorithm. Thus, we will later decompress it back, after receiving the response.

Please checkthis previous tutorial for an explanation on how to install the HTTP/2 sh2lib wrapper we are going to need. This wrapper works on top of the NGHTTP2 library and offers some higher level abstractions to make it easier to get started using the HTTP/2 protocol.

For a tutorial on how to do a HTTP/2 GET request, please checkhere. For a tutorial on how to use the ESP32 to decompress content compressed with the Brotli algorithm, please checkthis previous tutorial.

As can be seen here , there’s a HTTP request header called Accept-Encoding that allows the client to indicate to the server which content encoding it is capable of decoding [1].

This content encoding usually corresponds to a compression algorithm and Brotli is one of the supported algorithms [1]. So, if the client can work with Brotli, then this header should take the value “ br ” [1].

So, when setting up our HTTP/2 GET request, we will include the Accept-Encoding header with the value “ br “, so the server sends back the content encoded with the Brotli algorithm.This previous tutorial explains how to add headers to a HTTP/2 GET request, using the sh2lib library.

Naturally, not only the client but also the server must support the compression format.  Otherwise, even if the Accept-Encoding  header is specified by the client, the content won’t be compressed. And even if the server also supports this encoding, it may still choose to not compress the body of the response [1].

So, for testing, we will send the request to this endpoint, which supports the Brotli compression. As can be seen in figure 1, when we send a request using a web browser (in my case, Google Chrome), if the Accept-Encoding header has the “ br ” value included in the list of supported formats, then the server will use this compression algorithm.

Note that, as highlighted in figure 1, we can know if the server has used some encoding format by looking into the Content-Encoding response header. As shown, the server sent back that header with the value “ br “, thus indicating that Brotli was the compression format used.

Figure 2– Response to HTTP/2 GET request with Brotli compression.

You can also check from figure 1 that the expected answer, after decompression, is a JSON listing some libraries. This is what we should expect to obtain after running the code shown from the next sections on the ESP32.

The tests were performed using a DFRobot’s ESP32 module integrated in a  ESP32 development board .

The code

Includes and global variables

We will start our code by the library includes. As usual, we will need the WiFi.h  library, so we can connect the device to a WiFi network.

Besides that, we will need the sh2lib.h wrapper, so we can send the HTTP/2 request, and the decode.h module from the Brotli libraries, so we can decompress the content we will receive as response of our request.

Note that we need to enclose both these two last includes in an extern “C” block, for all the code to compile fine.

#include "WiFi.h"

extern "C" {
#include "sh2lib.h"
#include "decode.h"
}

We will also need to store the credentials of the WiFi network to which we are going to connect our ESP32. The needed credentials are the network name (SSID) and the password.

To finalize the global variables declaration, we will also need a Boolean flag that will be used to signalize when the request is finished. Naturally, we will initialize it to false and later, after the request is completed, we will set its value to true.

const char* ssid = "yourNetworkName";
const char* password =  "yourNetworkPassword";

bool request_finished = false;

Setup function

Moving on to the Arduino setup function, the code here will be the same we have been using in previous tutorials where we were doing HTTP/2 requests.

So, we start by opening a serial connection, to later output the results of our program. After that, we will connect the ESP32 to the WiFi network, using the previously declared credentials.

Finally, we will launch a FreeRTOS task that will be responsible for all the HTTP/2 related function calls.

void setup() {
  Serial.begin(115200);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }

  xTaskCreate(http2_task, "http2_task", (1024 * 32), NULL, 5, NULL);

}

Moving on to the implementation of the FreeRTOS task function, we will start by declaring a sh2lib_handle struct, which will be used as input of the next sh2lib function calls we will execute.

After that, we can connect to the server by calling the sh2lib_connect function, passing as first input the address of our handler, and as second input the server URL.

We will also perform an error check on the value returned by this function, to ensure the connection procedure was successful before we try to make the request.

struct sh2lib_handle hd;

if (sh2lib_connect(&hd, "https://api.cdnjs.com") != ESP_OK) {
    Serial.println("Error connecting to HTTP2 server");
    vTaskDelete(NULL);
}

Serial.println("Connected");

Now, since we want to indicate to the server that we can accept the content compressed with Brotli, we will need to set the “ accept-encoding ” header to the value “ br “.

As covered inthisprevious tutorial, we can specify the headers of our request by creating an array that will contain the name-value pairs representing those headers. Additionally to the “ accept-encoding ” header, we will also need to specify the HTTP/2 pseudo headers, which were also explained in detail on the mentioned tutorial.

const nghttp2_nv nva[] = { SH2LIB_MAKE_NV(":method", "GET"),
                           SH2LIB_MAKE_NV(":scheme", "https"),
                           SH2LIB_MAKE_NV(":authority", hd.hostname),
                           SH2LIB_MAKE_NV(":path", "/libraries/adapterjs"),
                           SH2LIB_MAKE_NV("accept-encoding", "br")
                          };

Then we will use the sh2lib_do_get_with_nv function to setup the request. This function receives as first input the address of our handle, as second the name-value array with the headers, as third the length of the name-value array and as fourth and final argument a callback function that will be invoked to handle the request response.

sh2lib_do_get_with_nv(&hd, nva, sizeof(nva) / sizeof(nva[0]), handle_get_response);

Since the previous function call only takes care of the request setup, we still need to call the sh2lib_execute periodically to execute the actual exchange of data with the server. This function simply receives as input the address of our handle.

As we did in previous tutorials, we will invoke this function periodically inside an infinite loop with a small delay between each iteration.

The loop will break when the request is finished, which will be signaled by the response handling function by setting the request_finished variable to true.

while (1) {

    if (sh2lib_execute(&hd) != ESP_OK) {
      Serial.println("Error in send/receive");
      break;
    }

    if (request_finished) {
      break;
    }

    vTaskDelay(10);
}

When the loop breaks, it means that the request is finished. So, after that, we will simply close the connection to the server by calling the sh2lib_free function and then we will delete the FreeRTOS task, which is no longer needed.

sh2lib_free(&hd);
Serial.println("Disconnected");

vTaskDelete(NULL);

The full FreeRTOS function implementation can be checked below.

void http2_task(void *args)
{
  struct sh2lib_handle hd;

  if (sh2lib_connect(&hd, "https://api.cdnjs.com") != ESP_OK) {
    Serial.println("Error connecting to HTTP2 server");
    vTaskDelete(NULL);
  }

  Serial.println("Connected");

  const nghttp2_nv nva[] = { SH2LIB_MAKE_NV(":method", "GET"),
                             SH2LIB_MAKE_NV(":scheme", "https"),
                             SH2LIB_MAKE_NV(":authority", hd.hostname),
                             SH2LIB_MAKE_NV(":path", "/libraries/adapterjs"),
                             SH2LIB_MAKE_NV("accept-encoding", "br")
                           };

  sh2lib_do_get_with_nv(&hd, nva, sizeof(nva) / sizeof(nva[0]), handle_get_response);

  while (1) {

    if (sh2lib_execute(&hd) != ESP_OK) {
      Serial.println("Error in send/receive");
      break;
    }

    if (request_finished) {
      break;
    }

    vTaskDelay(10);
  }

  sh2lib_free(&hd);
  Serial.println("Disconnected");

  vTaskDelete(NULL);
}

The response handling function

To finalize our code, we need to write the response handling function. As we have seen in previous tutorials, the signature of this function has the following signature:

int handle_get_response(struct sh2lib_handle *handle, const char *data, size_t len, int flags){
// handing function implementation
}

We can check if we have received data if the third argument (the length of the data) is greater than zero.

if (len > 0) {
// Handle received data
}

As seen before, if we receive data, we expect that it will be compressed with the Brotli algorithm. Note that, for simplicity, we are not checking the response headers to confirm this was the used encoding, but you should do it in a real application scenario.

So, the first thing we will do is declaring a byte buffer to store the decompressed data. We are going to declare an array that will be big enough to hold all the data that will be returned by the endpoint we will contact.

uint8_t buffer [4000];

As covered onthis previous tutorial, the function responsible for decompressing the content also receives the address of variable of type size_t , which will be used as an in and out parameter.

So, that variable should initially hold the size of the output data buffer (which we can obtain by using the sizeof operator) and then the decompress function will set that variable with the length of the actual decoded data.

In our case, the length of the decompressed content will be lesser than the total length the output buffer can hold, since we have declared an array that is bigger than what is actually needed. In a real case scenario, you may not know the actual size of the decompressed content (it may vary), so it makes sense to declare an array with the maximum length it can have.

size_t output_length = sizeof(buffer);

Then we will call the BrotliDecoderDecompress function, which will be responsible for doing the actual decompression.

As first input, this function receives the length of the compressed data. In our case, it corresponds to the third parameter that is passed to the HTTP/2 response handling function.

The second input corresponds to the buffer that holds the compressed data. In our case, it is the second argument of the response handling function.

Note that this argument is of type const char * and the decompress function argument is of type const uint8_t * , so we need to perform a cast when passing the variable.

As third argument, the function receives the address of the size_t variable we have declared before and initialized with the length of the output buffer.

As fourth and final argument, the BrotliDecoderDecompress function receives the output buffer, where it will write the decompressed content.

BrotliDecoderDecompress(
   len,
   (const uint8_t *)data,
   &output_length,
   buffer);

Then, after the decompression procedure is applied, we will print the resulting plain text data. We will do it using the printf function of the serial object and the %.*s format specifier, so we can directly use the output buffer and print it as a string.

Serial.printf("%.*sn", output_length, buffer);

The final callback function can be seen below. It also contains the handling of the stream closed event, which will set the request_finished variable to true, thus signaling the FreeRTOS function loop that the request is finished.

int handle_get_response(struct sh2lib_handle *handle, const char *data, size_t len, int flags)
{
  if (len > 0) {

    uint8_t buffer [4000];

    size_t output_length = sizeof(buffer);

    BrotliDecoderDecompress(
      len,
      (const uint8_t *)data,
      &output_length,
      buffer);

    Serial.printf("%.*sn", output_length, buffer);
  }

  if (flags == DATA_RECV_RST_STREAM) {
    request_finished = true;
    Serial.println("STREAM CLOSED");
  }
  return 0;
}

The final code

The final code can be seen below.

#include "WiFi.h"

extern "C" {
#include "sh2lib.h"
#include "decode.h"
}

const char* ssid = "yourNetworkName";
const char* password =  "yourNetworkPassword";

bool request_finished = false;

int handle_get_response(struct sh2lib_handle *handle, const char *data, size_t len, int flags)
{
  if (len > 0) {

    uint8_t buffer [4000];

    size_t output_length = sizeof(buffer);

    BrotliDecoderDecompress(
      len,
      (const uint8_t *)data,
      &output_length,
      buffer);

    Serial.printf("%.*sn", output_length, buffer);
  }

  if (flags == DATA_RECV_RST_STREAM) {
    request_finished = true;
    Serial.println("STREAM CLOSED");
  }
  return 0;
}

void http2_task(void *args)
{
  struct sh2lib_handle hd;

  if (sh2lib_connect(&hd, "https://api.cdnjs.com") != ESP_OK) {
    Serial.println("Error connecting to HTTP2 server");
    vTaskDelete(NULL);
  }

  Serial.println("Connected");

  const nghttp2_nv nva[] = { SH2LIB_MAKE_NV(":method", "GET"),
                             SH2LIB_MAKE_NV(":scheme", "https"),
                             SH2LIB_MAKE_NV(":authority", hd.hostname),
                             SH2LIB_MAKE_NV(":path", "/libraries/adapterjs"),
                             SH2LIB_MAKE_NV("accept-encoding", "br")
                           };

  sh2lib_do_get_with_nv(&hd, nva, sizeof(nva) / sizeof(nva[0]), handle_get_response);

  while (1) {

    if (sh2lib_execute(&hd) != ESP_OK) {
      Serial.println("Error in send/receive");
      break;
    }

    if (request_finished) {
      break;
    }

    vTaskDelay(10);
  }

  sh2lib_free(&hd);
  Serial.println("Disconnected");

  vTaskDelete(NULL);
}

void setup() {
  Serial.begin(115200);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }

  xTaskCreate(http2_task, "http2_task", (1024 * 32), NULL, 5, NULL);

}

void loop() {
  vTaskDelete(NULL);
}

Testing the code

To test the code, simply compile it and upload it to your ESP32, using the Arduino IDE. After the procedure finishes, open the serial monitor. You should get an output similar to figure 2, which shows the decompressed content.

Note that this shows just part of the response since the API sends the JSON without any newlines, but the whole response was correctly decompressed.

Figure 2– Printing the decompressed content.

References

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding

techtutorialsx

责编内容by:techtutorialsx阅读原文】。感谢您的支持!

您可能感兴趣的

Go 语言 HTTP 请求超时入门 在分布式系统中,超时是基本可靠性概念之一。就像这条 tweet 中提到的,它可以缓和分布式系统中不可避免出现的失败所带来的影响。 问题 如何条件性地模拟 504 http.StatusGatewayTimeout 响应。...
The HTTP Archive got a huge upgrade 💪 Announcements rviscomi 2018-03-27 02:00:25 UTC #1 Today we’re excited to be graduating the website f...
What is HTTP Part II – Underlying Protocols Last week in part one of this series , we took the 50,000 foot view of the HTTP protocol. HTTP defines the structure of...
Express module error while listening http server i... I have created a nodejs http server var http = require("http"); var url = require("url"); var express = require('exp...
静态代码块详解(原出处:http://versioneye.iteye.com/blog/11295... 一 般情况下,如果有些代码必须在项目启动的时候就执行的时候,需要使用静态代码块,这种代码是主动执行的;需要在项目启动的时候就初始化,在不创建对象的情 况下,其他程序来调用的时候,需要使用静态方法,这种代码是被动执行的. 静态方法在类加...