Ad

Why Does S3.putObject Throw ERR_INVALID_ARG_TYPE For Nodejs Buffer?

I am writing a lambda function in nodejs for uploading an image to an S3 bucket.

In my first lambda function I have written this inline (directly in AWS Lambda editor) and it works perfectly fine. The relevant code is:

let decodedImage = Buffer.from(encodedImage, 'base64');
    console.log(decodedImage);
    console.log(typeof decodedImage);
    console.log(Object.prototype.toString.call(decodedImage));

    try {
        await writeToBucket(decodedImage, 'test.png', 'image/png');
    } catch (err) {
        console.log(err);
    } 

async function writeToBucket(data, dbFileId, mimeType) {
    return new Promise((resolve, reject) => {

        var s3 = new AWS.S3();
        const params = {
            Bucket : S3_BUCKET,
            Key : dbFileId,
            Body : data,
            ACL: 'public-read',
            ContentType: mimeType
        }

        s3.putObject(params, function(err, dataString) {
            if (err) {
                reject([503, err]);
            } else {
                resolve();
            }
        });
    });

The three logs gives:

<Buffer 89 50 4e 47 ... >
object
[object UInt8Array]

So, as said, the above works fine and the image is written to bucket.

Now, I have a second Lambda function, which I write locally and upload to Lambda after building with webpack, zipping and uploading. The code is the same in both lambdas (as above), and the console output is also identical. But in the second scenario the upload fails, withe the following error:

{ NetworkingError: The "chunk" argument must be one of type string or Buffer. Received type object
at ClientRequest.end (_http_outgoing.js:690:13)
at features.constructor.writeBody (/var/runtime/node_modules/aws-sdk/lib/http/node.js:137:14)
at features.constructor.handleRequest (/var/runtime/node_modules/aws-sdk/lib/http/node.js:105:12)
at executeSend (/var/runtime/node_modules/aws-sdk/lib/event_listeners.js:342:29)
at Request.SEND (/var/runtime/node_modules/aws-sdk/lib/event_listeners.js:356:9)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:102:18)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
message:
'The "chunk" argument must be one of type string or Buffer. Received type object',
code: 'NetworkingError',
region: 'eu-west-1',
hostname: 's3.eu-west-1.amazonaws.com',
retryable: true,
time: 2019-08-26T14:25:23.006Z } 

As far as I can see the Body payload is indeed of type Buffer and I can't figure out what is the difference? Can it be Webpack (or Babel) that messes up something? Or can it have something to do with Nodejs.Buffer? I run Node.js 10.x in both Lambdas.

My Webpack setup is this:

const path = require('path');
require("@babel/register");
require("@babel/plugin-transform-runtime");

const config = {
    mode: 'development',
  entry: './index.js',

  output: {
    path: path.resolve(__dirname, 'dist'),
    filename: 'index.js',
    library: 'index',
    libraryTarget: 'commonjs2'
  },
  // Loaders
  module: {
    rules : [
      { test: /\.js$/, exclude: /node_modules/, loader: "babel-loader" },
      { test: /\.css$/, use: ['style-loader', 'css-loader']}
    ]
  },
  // Plugins
  plugins: [],
  optimization:{
    minimize: false
  },
  externals: {
    'aws-sdk': 'aws-sdk'
  }
};
// Exports
module.exports = config; 

And package.json dependencies:

"dependencies": {
    "@babel/register": "~7.5.5",
    "@babel/runtime": "~7.5.5",
    "ajv": "~6.10.2",
    "ajv-errors": "~1.0.1",
    "css-loader": "~3.2.0",
    "style-loader": "~1.0.0",
    "uuid": "~3.3.3"
  },
  "devDependencies": {
    "@babel/cli": "~7.5.5",
    "@babel/core": "~7.5.5",
    "@babel/node": "~7.5.5",
    "@babel/plugin-transform-runtime": "~7.5.5",
    "@babel/preset-env": "~7.5.5",
    "aws-sdk": "~2.517.0",
    "babel-loader": "~8.0.6",
    "cross-var-no-babel": "~1.2.0",
    "mocha": "^6.1.4",
    "webpack": "~4.33.0",
    "webpack-cli": "~3.3"
  }
Ad

Answer

This error is confusing since your code is indeed using a Buffer as Body and there's not any chunk argument visible, so that must be internal to the SDK. In isolation, the assumption would be that the SDK isn't handling something correctly, and is throwing an exception much later than should be expected (e.g. not as a direct result from calling s3.putObject() but rather later, after the actual "work" has started).

I had a nagging thought that I read somewhere about a problem with the AWS SDK when trying to optimize its size for distribution... but I couldn't find what I was looking for, plus you have it in externals so maybe that's unrelated.

But I did find where Bundling Applications with Webpack in the AWS SDK for JavaScript Developer Guide for SDK v2 mentions something potentially relevant:

You can use webpack to generate bundles that run in Node.js by specifying it as a target in the configuration.

target: "node"

The webpack docs mention that this "...will compile for usage in a Node.js-like environment (uses Node.js require to load chunks and not touch any built in modules like fs or path)". It isn't obvious to me if this explanation contains a hidden nugget that accounts for the reason why not including it breaks the SDK, or if there are other things going on... but specifying this appears to be the fix.

If I were to speculate, my guess is that this works by disabling inclusion of a Buffer polyfill that is otherwise added by webpack.

Ad
source: stackoverflow.com
Ad