Posts Tagged ‘TensorFlowJS’
[DevoxxUK2024] Is It (F)ake?! Image Classification with TensorFlow.js by Carly Richmond
Carly Richmond, a Principal Developer Advocate at Elastic, captivated the DevoxxUK2024 audience with her engaging exploration of image classification using TensorFlow.js. Inspired by her love for the Netflix show Is It Cake?, Carly embarked on a project to build a model distinguishing cakes disguised as everyday objects from their non-cake counterparts. Despite her self-professed lack of machine learning expertise, Carly’s journey through data gathering, pre-trained models, custom model development, and transfer learning offers a relatable and insightful narrative for developers venturing into AI-driven JavaScript applications.
Gathering and Preparing Data
Carly’s project begins with the critical task of data collection, a foundational step in machine learning. To source images of cakes resembling other objects, she leverages Playwright, a JavaScript-based automation framework, to scrape images from bakers’ websites and Instagram galleries. For non-cake images, Carly utilizes the Unsplash API, which provides royalty-free photos with a rate-limited free tier. She queries categories like reptiles, candles, and shoes to align with the deceptive cakes from the show. However, Carly acknowledges limitations, such as inadvertently including biscuits or company logos in the dataset, highlighting the challenges of ensuring data purity with a modest set of 367 cake and 174 non-cake images.
Exploring Pre-Trained Models
To avoid building a model from scratch, Carly initially experiments with TensorFlow.js’s pre-trained models, Coco SSD and MobileNet. Coco SSD, trained on the Common Objects in Context (COCO) dataset, excels in object detection, identifying bounding boxes and classifying objects like cakes with reasonable accuracy. MobileNet, designed for lightweight classification, struggles with Carly’s dataset, often misclassifying cakes as cups or ice cream due to visual similarities like frosting. CORS issues further complicate browser-based MobileNet deployment, prompting Carly to shift to a Node.js backend, where she converts images into tensors for processing. These experiences underscore the trade-offs between model complexity and practical deployment.
Building and Refining a Custom Model
Undeterred by initial setbacks, Carly ventures into crafting a custom convolutional neural network (CNN) using TensorFlow.js. She outlines the CNN’s structure, which includes convolution layers to extract features, pooling layers to reduce dimensionality, and a softmax activation for binary classification (cake vs. not cake). Despite her efforts, the model’s accuracy languishes at 48%, plagued by issues like tensor shape mismatches and premature tensor disposal. Carly candidly admits to errors, such as mislabeling cakes as non-cakes, illustrating the steep learning curve for non-experts. This section of her talk resonates with developers, emphasizing perseverance and the iterative nature of machine learning.
Leveraging Transfer Learning
Recognizing the limitations of her dataset and custom model, Carly pivots to transfer learning, using MobileNet’s feature vectors as a foundation. By adding a custom classification head with ReLU and softmax layers, she achieves a significant improvement, with accuracy reaching 100% by the third epoch and correctly classifying 319 cakes. While not perfect, this approach outperforms her custom model, demonstrating the power of leveraging pre-trained models for specialized tasks. Carly’s comparison of human performance—90% accuracy by the DevoxxUK audience versus her model’s results—adds a playful yet insightful dimension, highlighting the gap between human intuition and machine precision.
Links:
[NodeCongress2021] Machine Learning in Node.js using Tensorflow.js – Shivay Lamba
The fusion of machine learning capabilities with server-side JavaScript environments opens intriguing avenues for developers seeking to embed intelligent features directly into backend workflows. Shivay Lamba, a versatile software engineer proficient in DevOps, machine learning, and full-stack paradigms, illuminates this intersection through his examination of TensorFlow.js within Node.js ecosystems. As an open-source library originally developed by the Google Brain team, TensorFlow.js democratizes access to sophisticated neural networks, allowing practitioners to train, fine-tune, and infer models without forsaking the familiarity of JavaScript syntax.
Shivay’s narrative commences with the foundational allure of TensorFlow.js: its seamless portability across browser and Node.js contexts, underpinned by WebGL acceleration for tensor operations. This universality sidesteps the silos often encountered in traditional ML stacks, where Python dominance necessitates cumbersome bridges. In Node.js, the library harnesses native bindings to leverage CPU/GPU resources efficiently, enabling tasks like image classification or natural language processing to unfold server-side. Shivay emphasizes practical onboarding—install via npm, import tf, and instantiate models—transforming abstract algorithms into executable logic.
Consider a sentiment analysis endpoint: load a pre-trained BERT variant, preprocess textual inputs via tokenizers, and yield probabilistic outputs—all orchestrated in asynchronous handlers to maintain Node.js’s non-blocking ethos. Shivay draws from real-world deployments, where such integrations power recommendation engines or anomaly detectors in e-commerce pipelines, underscoring the library’s scalability for production loads.
Streamlining Model Deployment and Inference
Deployment nuances emerge as Shivay delves into optimization strategies. Quantization shrinks model footprints, slashing latency for edge inferences, while transfer learning adapts pre-trained architectures to domain-specific corpora with minimal retraining epochs. He illustrates with a convolutional neural network for object detection: convert ONNX formats to TensorFlow.js via converters, bundle with webpack for serverless functions, and expose via Express routes. Monitoring integrates via Prometheus metrics, tracking inference durations and accuracy drifts.
Challenges abound—memory constraints in containerized setups demand careful tensor management, mitigated by tf.dispose() invocations. Shivay advocates hybrid approaches: offload heavy training to cloud TPUs, reserving Node.js for lightweight inference. Community extensions, like @tensorflow/tfjs-node-gpu, amplify throughput on NVIDIA hardware, aligning with Node.js’s event-driven architecture.
Shivay’s exposition extends to ethical considerations: bias audits in datasets ensure equitable outcomes, while federated learning preserves privacy in distributed training. Through these lenses, TensorFlow.js transcends novelty, evolving into a cornerstone for ML-infused Node.js applications, empowering creators to infuse intelligence without infrastructural overhauls.