Archive for the ‘en-US’ Category

Chayim Kirshen, a veteran of the startup ecosystem and client manager at Redis, captivated audiences at Devoxx Poland 2022 with a dynamic exploration of real-time data pipeline challenges. Drawing from his experience with high-stakes environments, including a 2010 stock exchange meltdown, Chayim outlined strategies to ensure data integrity and performance in large-scale systems. His talk provided actionable insights for developers, emphasizing the importance of storing raw data, parsing in real time, and leveraging technologies like Redis to address data inconsistencies.

The Perils of Unclean Data

Chayim began with a stark reality: data is rarely clean. Recounting a 2010 incident where hackers compromised a major stock exchange’s API, he highlighted the cascading effects of unreliable data on real-time markets. Data pipelines face issues like inconsistent formats (CSV, JSON, XML), changing sources (e.g., shifting API endpoints), and service reliability, with modern systems often tolerating over a thousand minutes of downtime annually. These challenges disrupt real-time processing, critical for applications like stock exchanges or ad bidding networks requiring sub-100ms responses. Chayim advocated treating data as programmable code, enabling developers to address issues systematically rather than reactively.

Building Robust Data Pipelines

To tackle these issues, Chayim proposed a structured approach to data pipeline design. Storing raw data indefinitely—whether in S3, Redis, or other storage—ensures a fallback for reprocessing. Parsing data in real time, using defined schemas, allows immediate usability while preserving raw inputs. Bulk changes, such as SQL bulk inserts or Redis pipelines, reduce network overhead, critical for high-throughput systems. Chayim emphasized scheduling regular backfills to re-import historical data, ensuring consistency despite source changes. For example, a stock exchange’s ticker symbol updates (e.g., Fitbit to Google) require ongoing reprocessing to maintain accuracy. Horizontal scaling, using disposable nodes, enhances availability and resilience, avoiding single points of failure.

Real-Time Enrichment and Redis Integration

Data enrichment, such as calculating stock bid-ask spreads or market cap changes, should occur post-ingestion to avoid slowing the pipeline. Chayim showcased Redis, particularly its Gears and JSON modules, for real-time data processing. Redis acts as a buffer, storing raw JSON and replicating it to traditional databases like PostgreSQL or MySQL. Using Redis Gears, developers can execute functions within the database, minimizing network costs and enabling rapid enrichment. For instance, calculating a stock’s daily percentage change can run directly in Redis, streamlining analytics. Chayim highlighted Python-based tools like Celery for scheduling backfills and enrichments, allowing asynchronous processing and failure retries without disrupting the main pipeline.

Scaling and Future-Proofing

Chayim stressed horizontal scaling to distribute workloads geographically, placing data closer to users for low-latency access, as seen in ad networks. By using Redis for real-time writes and offloading to workers via Celery, developers can manage millions of daily entries, such as stock ticks, without performance bottlenecks. Scheduled backfills address data gaps, like API schema changes (e.g., integer to string conversions), by reprocessing raw data. This approach, combined with infrastructure-as-code tools like Terraform, ensures scalability and adaptability, allowing organizations to focus on business logic rather than data management overhead.

Links:

• Redis company website

At Devoxx Poland 2022, Paweł Piekut, a seasoned software developer at Bosch, delivered an insightful presentation on the migration of their e-bike cloud platform from a private cloud to a public cloud environment. Drawing from his expertise in Java, Kotlin, and .NET, Paweł narrated the intricate journey of transitioning a complex IoT ecosystem, highlighting the technical challenges, strategic decisions, and lessons learned. His talk offered a practical roadmap for organizations navigating the complexities of cloud migration, emphasizing the balance between innovation, scalability, and compliance.

Navigating the Private Cloud Landscape

Paweł began by outlining the initial deployment of Bosch’s e-bike cloud on a private cloud developed internally by the company’s IT group. This proprietary platform, designed to support the e-bike ecosystem, facilitated communication between hardware components—such as drive units, batteries, and controllers—and the mobile app, which interfaced with the cloud. The cloud served multiple stakeholders, including factories for device flashing, manufacturers for configuration, authorized services for diagnostics, and end-users for features like activity tracking and bike locking. However, the private cloud faced significant limitations. Scalability was constrained, requiring manual capacity requests and investments, which hindered agility. Downtimes were frequent, acceptable for development but untenable for production. Additionally, the platform’s bespoke nature made it challenging to hire experienced talent and limited developer engagement due to its lack of market-standard tools.

Despite these drawbacks, the private cloud offered advantages. Its deployment within Bosch’s secure network ensured high performance and simplified compliance with data privacy regulations, critical for an international product subject to data localization laws. Costs were predictable, and the absence of vendor lock-in, thanks to open-source frameworks, provided flexibility. However, the need for modern scalability and developer-friendly tools drove the decision to explore public cloud solutions, with Amazon Web Services (AWS) selected for its robust support.

The Hybrid Cloud Conundrum

Transitioning to a hybrid cloud model introduced a blend of private and public cloud environments, creating new challenges. Bosch’s internal policy of “on-transit data” required data processed in the public cloud to be returned to the private cloud, necessitating complex and secure data transfers. While AWS Direct Connect facilitated this, the hybrid setup led to operational complexities. Only select services ran on AWS, causing a divide among developers eager to work with widely recognized public cloud tools. Technical issues, such as Kafka’s inaccessibility from the private cloud, required significant effort to resolve. Error tracing across clouds was cumbersome, with Splunk used in the private cloud and Elasticsearch in the public cloud, complicating root-cause analysis. The simultaneous migration of Jenkins added further complexity, with duplicated jobs and confusing configurations.

Despite these hurdles, the hybrid model offered benefits. It allowed Bosch to leverage the private cloud’s security for sensitive data while tapping into the public cloud’s scalability for peak loads. This setup supported disaster recovery and compliance with data localization requirements. However, the on-transit data concept proved overly complex, leading to dissatisfaction and prompting a strategic shift toward a cloud-first approach, prioritizing public cloud deployment unless justified otherwise.

Embracing the Public Cloud

The full migration to AWS marked a pivotal phase, divided into three stages. First, the team focused on exploration and training to master AWS products and the pay-as-you-go pricing model, which made every developer accountable for costs. This stage emphasized understanding managed versus unmanaged services, such as Kubernetes and Kafka, and ensuring backup compatibility across clouds. The second stage involved building new applications on AWS, addressing unknowns and ensuring secure communication with external systems. Finally, existing services were migrated from private to public cloud, starting with development and progressing to production. Throughout, the team maintained services in both environments, managing separate repositories and addressing critical bugs, such as Log4j vulnerabilities, across both.

To mitigate vendor lock-in, Bosch adopted a cloud-agnostic approach, using Terraform for infrastructure-as-code instead of AWS-specific CloudFormation. While tools like S3 and DynamoDB were embraced for their market-leading performance, backups were standardized to ensure portability. The public cloud’s vast community, extensive documentation, and readily available resources reduced knowledge silos and enhanced developer satisfaction, making the migration a transformative step for innovation and agility.

Lessons for Cloud Migration

Paweł’s experience underscores the importance of aligning cloud strategy with organizational needs. The public cloud’s immediate resource availability and developer-friendly tools accelerated development, but required careful cost management. Hybrid cloud offered flexibility but introduced complexity, particularly with data transfers. Private cloud provided security and control but lacked scalability. Paweł emphasized defining precise requirements—budget, priorities, and compliance—before choosing a cloud model. Startups may favor public clouds for agility, while regulated industries might opt for private or hybrid solutions to prioritize data security and network performance. This strategic clarity ensures a successful migration tailored to business goals.

Links:

• Bosch company website

Philipp Krenn, a developer advocate at Elastic, captivated audiences at Devoxx Poland 2022 with a witty and incisive exploration of semantic versioning (SemVer). Drawing from Elastic’s experiences with Elasticsearch, Philipp dissected the nuances of versioning, revealing why SemVer often ignites passionate debates. His talk navigated the ambiguities of defining APIs, the complexities of breaking changes, and the cultural dynamics of open-source versioning, offering a pragmatic lens for developers grappling with version management.

Decoding Semantic versioning

Philipp introduced SemVer, as formalized on semver.org, with its major version structure, where patch fixes bugs, minor adds features, and major introduces breaking changes. This simplicity, however, belies complexity in practice. He posed a sorting challenge with version strings like alpha.-, 2.-, and 11.-, illustrating SemVer’s arcane precedence rules, humorously cautioning against such obfuscation unless “trolling users.” Philipp noted that SemVer’s focus on APIs raises fundamental questions: what constitutes an API? For Elasticsearch, the REST API is sacrosanct, warranting major version bumps for changes, whereas plugin APIs, exposing internal Java packages, tolerate frequent breaks, sparking user frustration when plugins fail.

The Ambiguity of Breaking Changes

The definition of a breaking change varies by perspective, Philipp argued. Upgrading a supported JDK version, for instance, divides opinions—some view it as a system-altering break, others as an implementation detail. Security fixes further muddy the waters, as seen in Elastic’s handling of unintended insecure usage, where API “fixes” disrupted user workflows. Philipp cited the Log4j2 vulnerability, where maintainers supported multiple JDK versions across minor releases, avoiding major version increments. Accidental breaks, common in open-source projects, and asymmetric feature additions—easy to add, hard to remove—compound SemVer’s challenges, often leading to user distrust when expectations misalign.

Cultural and Practical Dilemmas

Philipp explored why SemVer debates are so heated, attributing it to differing interpretations of “correct” versioning. He critiqued version ranges, prevalent in npm but rare in Java, for introducing instability due to transitive dependency updates, advocating for tools like Dependabot to manage updates explicitly. Experimental APIs, marked as unstable, offer an escape hatch for breaking changes without major version bumps, though they demand diligent release note scrutiny. Pre-1.0 versions, dubbed the “Wild West,” lack SemVer guarantees, enabling unfettered changes but risking user confusion. Philipp contrasted SemVer with alternatives like calendar versioning, used by Ubuntu, noting its decline as SemVer dominates modern ecosystems.

Links:

• Elastic company website
• Semantic Versioning specification

Jacek Bzdak and Beata Strack, software engineers at Google Poland, delivered an engaging session at Devoxx Poland 2022, exploring the intricacies of optimizing Google’s planet-scale computing infrastructure. Their talk focused on achieving efficiency in a distributed system spanning global data centers, emphasizing resource utilization, auto-scaling, and operational strategies. By sharing insights from Google’s internal cloud and Autopilot system, Jacek and Beata provided a blueprint for enhancing service performance while navigating the complexities of large-scale computing.

Defining Efficiency in a Global Fleet

Beata opened by framing Google’s data centers as a singular “planet-wide computer,” where efficiency translates to minimizing operational costs—servers, CPU, memory, data centers, and electricity. Key metrics like fleet-wide utilization, CPU/RAM allocation, and growth rate serve as proxies for these costs, though they are imperfect, often masking quality issues like inflated memory usage. Beata stressed that efficiency begins at the service level, where individual jobs must optimize resource consumption, and extends to the fleet through an ecosystem that maximizes resource sharing. This dual approach ensures that savings at the micro level scale globally, a principle applicable even to smaller organizations.

Auto-Scaling: Balancing Utilization and Reliability

Jacek, a member of Google’s Autopilot team, delved into auto-scaling, a critical mechanism for achieving high utilization without compromising reliability. Autopilot’s vertical scaling adjusts resource limits (CPU/memory) for fixed replicas, while horizontal scaling modifies replica counts. Jacek presented data from an Autopilot paper, showing that auto-scaled services maintain memory slack below 20% for median cases, compared to over 60% for manually managed services. Crucially, automation reduces outage risks by dynamically adjusting limits, as demonstrated in a real-world case where Autopilot preempted a memory-induced crash. However, auto-scaling introduces complexity, particularly feedback loops, where overzealous caching or load shedding can destabilize resource allocation, requiring careful integration with application-specific metrics.

Java-Specific Challenges in Auto-Scaling

The talk transitioned to language-specific hurdles, with Jacek highlighting Java’s unique challenges in auto-scaling environments. Just-in-Time (JIT) compilation during application startup spikes CPU usage, complicating horizontal scaling decisions. Memory management poses further issues, as Java’s heap size is static, and out-of-memory errors may be masked by garbage collection (GC) thrashing, where excessive CPU is devoted to GC rather than request handling. To address this, Google sets static heap sizes and auto-scales non-heap memory, though Jacek envisioned a future where Java aligns with other languages, eliminating heap-specific configurations. These insights underscore the need for language-aware auto-scaling strategies in heterogeneous environments.

Operational Strategies for Resource Reclamation

Beata concluded by discussing operational techniques like overcommit and workload colocation to reclaim unused resources. Overcommit leverages the low probability of simultaneous resource spikes across unrelated services, allowing Google to pack more workloads onto machines. Colocating high-priority serving jobs with lower-priority batch workloads enables resource reclamation, with batch tasks evicted when serving jobs demand capacity. A 2015 experiment demonstrated significant machine savings through colocation, a concept influencing Kubernetes’ design. These strategies, combined with auto-scaling, create a robust framework for efficiency, though they demand rigorous isolation to prevent interference between workloads.

Links:

• Google company website

At Devoxx Poland 2022, Piotr Bochyński, a seasoned cloud native expert at SAP, shared a compelling narrative on transitioning customers and internal teams from a Cloud Foundry-based platform to Kubernetes. His presentation illuminated the strategic imperatives, technical challenges, and practical solutions that defined SAP’s journey toward a multi-cloud Kubernetes ecosystem. By leveraging open-source projects like Kyma and Gardener, Piotr’s team addressed the limitations of their legacy platform, fostering developer productivity and operational scalability. His insights offer valuable lessons for organizations contemplating a similar migration.

Understanding Platform as a Service

Piotr began by contextualizing Platform as a Service (PaaS), a model that abstracts infrastructure complexities, allowing developers to focus on application development. Unlike Infrastructure as a Service (IaaS), which provides raw virtual machines, PaaS delivers managed runtimes, middleware, and automation, accelerating time-to-market. However, this convenience comes with trade-offs, such as reduced control and potential vendor lock-in, often tied to opinionated frameworks like the 12-factor application methodology. Piotr highlighted SAP’s initial adoption of Cloud Foundry, an open-source PaaS, to avoid vendor dependency while meeting multi-cloud requirements driven by legal and business needs, particularly in sectors like banking. Yet, Cloud Foundry’s constraints, such as single HTTP port exposure and reliance on outdated technologies like BOSH, prompted SAP to explore Kubernetes as a more flexible alternative.

Kubernetes: A Platform for Platforms

Kubernetes, as Piotr elucidated, is not a traditional PaaS but a container orchestration framework that serves as a foundation for building custom platforms. Its declarative API and extensibility distinguish it from predecessors, enabling consistent management of diverse resources like deployments, namespaces, and custom objects. Piotr illustrated this with the thermostat analogy: developers declare a desired state (e.g., 22 degrees), and Kubernetes controllers reconcile the actual state to match it. This pattern, applied uniformly across resources, empowers developers to extend Kubernetes with custom controllers, such as a hypothetical thermostat resource. The Kyma project, an open-source initiative led by SAP, builds on this extensibility, providing opinionated building blocks like Istio-based API gateways, NATS eventing, and serverless functions to bridge the gap between raw Kubernetes and a developer-friendly PaaS.

Overcoming Migration Challenges

The migration to Kubernetes presented multifaceted challenges, from technical complexity to cultural adoption. Piotr emphasized the steep learning curve associated with Kubernetes’ vast resource set, compounded by additional components like Prometheus and Istio. To mitigate this, SAP employed Kyma to abstract complexities, offering simplified resources like API rules that encapsulate Istio configurations for secure service exposure. Another hurdle was ensuring multi-cloud compatibility. SAP’s Gardener project, a managed Kubernetes solution, addressed this by providing a consistent, Kubernetes-compliant layer across providers like AWS, Azure, and Google Cloud. Piotr also discussed operational scalability, managing thousands of clusters for hundreds of teams. By applying the Kubernetes controller pattern, SAP automated cluster provisioning, upgrades, and security patching, reducing manual intervention and ensuring reliability.

Lessons from the Journey

Reflecting on the migration, Piotr candidly shared missteps that shaped SAP’s approach. Early attempts to shield users from Kubernetes’ complexity by mimicking Cloud Foundry’s API failed, as developers craved direct control over Kubernetes resources. Similarly, restricting cluster admin roles to prevent misconfigurations stifled innovation, leading SAP to grant greater flexibility. Some technology choices, like the Service Catalog project, proved inefficient, underscoring the importance of aligning with Kubernetes’ operator pattern. License changes in tools like Grafana also necessitated pivots, highlighting the need for vigilance in open-source dependencies. Piotr’s takeaways resonate broadly: Kubernetes is a long-term investment, requiring a balance of opinionated tooling and developer freedom, with automation as a cornerstone for scalability.

Links:

• SAP company website
• Kyma project on GitHub
• Gardener project on GitHub

Piotr Przybył, a seasoned software gardener at AtomicJar, captivated the audience at Devoxx Poland 2022 with a comprehensive deep dive into the new features and enhancements introduced in Java 17 and 18. His presentation, rich with technical insights and practical demonstrations, explored key updates that empower developers to write more robust, maintainable, and efficient code. Piotr’s engaging style, peppered with humor and real-world examples, provided a clear roadmap for leveraging these advancements in modern Java development.

Sealed Classes for Controlled Inheritance

One of the standout features of Java 17 is sealed classes, introduced as JEP 409. Piotr explained how sealed classes allow developers to restrict which classes or interfaces can extend or implement a given type, offering fine-grained control over inheritance. This is particularly useful for library maintainers who want to prevent unintended code reuse while allowing specific extensions. By using the sealed keyword and a permits clause, developers can define a closed set of subclasses, with options to mark them as final, sealed, or non-sealed. Piotr’s demo illustrated this with a library type hierarchy, showing how sealed classes enhance code maintainability and prevent misuse through inheritance.

Enhanced Encapsulation and UTF-8 by Default

Java 17’s JEP 403 strengthens encapsulation by removing illegal reflective access, a change Piotr humorously likened to “closing the gates to reflection demons.” Previously, developers could bypass encapsulation using setAccessible(true), but Java 17 enforces stricter access controls, requiring code fixes or the use of --add-opens flags for legacy systems. Additionally, Java 18’s JEP 400 sets UTF-8 as the default charset for I/O operations, resolving discrepancies across platforms. Piotr demonstrated how to handle encoding issues, advising developers to explicitly specify charsets to ensure compatibility, especially for Windows users.

Deprecating Finalization and Introducing Simple Web Server

Java 18’s JEP 421 marks the deprecation of the finalize method for removal, signaling the end of a problematic mechanism for resource cleanup. Piotr’s demo highlighted the non-deterministic nature of finalization, advocating for try-with-resources as a modern alternative. He also showcased Java 18’s simple web server (JEP 408), a lightweight tool for serving static files during development or testing. Through a programmatic example, Piotr demonstrated how to start a server on port 9000 and dynamically modify CSS files, emphasizing its utility for quick prototyping.

Pattern Matching for Switch and Foreign Function API

Piotr explored Java 18’s pattern matching for switch (JEP 420), a preview feature that enhances switch statements and expressions. This feature supports null handling, guarded patterns, and type-based switching, eliminating the need for cumbersome if-else checks. His demo showed how to switch over objects, handle null cases, and use guards to refine conditions, making code more concise and readable. Additionally, Piotr introduced the Foreign Function and Memory API (JEP 419), an incubator module for safe, efficient interoperation with native code. He demonstrated allocating off-heap memory and calling C functions, highlighting the API’s thread-safety and scope-bound memory management.

Random Generators and Deserialization Filters

Java 17’s JEP 356 introduces enhanced pseudo-random number generators, offering a unified interface for various random number implementations. Piotr’s demo showcased switching between generators like Random, SecureRandom, and ThreadLocalRandom, simplifying random number generation for diverse use cases. Java 17 also improves deserialization filters (JEP 415), allowing per-stream customization to enhance security against malicious data. These updates, combined with other enhancements like macOS Metal rendering and larger G1 heap regions, underscore Java’s commitment to performance and security.

Links:

• AtomicJar company website
• Piotr Przybył’s website

At Devoxx Poland 2022, Bartosz de Boulange, a Java developer at BGŻ BNP Paribas, Poland’s national development bank, delivered an insightful presentation on Hibernate Search, a powerful tool that seamlessly integrates traditional Object-Relational Mapping (ORM) with NoSQL databases like Elasticsearch. Bartosz’s talk focused on enabling full-text search capabilities within SQL-based applications, offering a practical solution for developers seeking to enhance search functionality without migrating entirely to a NoSQL ecosystem. Through a blend of theoretical insights and hands-on coding demonstrations, he illustrated how Hibernate Search can address complex search requirements in modern applications.

The Power of Full-Text Search

Bartosz began by addressing the challenge of implementing robust search functionality in applications backed by SQL databases. For instance, in a bookstore application, users might need to search for specific phrases within thousands of reviews. Traditional SQL queries, such as LIKE statements, are often inadequate for such tasks due to their limited ability to handle complex text analysis. Hibernate Search solves this by enabling full-text search, which includes character filtering, tokenization, and normalization. These features allow developers to remove irrelevant characters, break text into searchable tokens, and standardize data for efficient querying. Unlike native SQL full-text search capabilities, Hibernate Search offers a more streamlined and scalable approach, making it ideal for applications requiring sophisticated search features.

Integrating Hibernate with Elasticsearch

The core of Bartosz’s presentation was a step-by-step guide to integrating Hibernate Search with Elasticsearch. He outlined five key steps: creating JPA entities, adding Hibernate Search dependencies, annotating entities for indexing, configuring fields for NoSQL storage, and performing initial indexing. By annotating entities with @Indexed, developers can create indexes in Elasticsearch at application startup. Fields are annotated as @FullTextField for tokenization and search, @KeywordField for sorting, or @GenericField for basic querying. Bartosz emphasized the importance of the @FullTextField, which enables advanced search capabilities like fuzzy matching and phrase queries. His live coding demo showcased how to set up a Docker Compose file with MySQL and Elasticsearch, configure the application, and index a bookstore’s data, demonstrating the ease of integrating these technologies.

Scalability and Synchronization Challenges

A significant advantage of using Elasticsearch with Hibernate Search is its scalability. Unlike Apache Lucene, which is limited to a single node and suited for smaller projects, Elasticsearch supports distributed data across multiple nodes, making it ideal for enterprise applications. However, Bartosz highlighted a key challenge: synchronization between SQL and NoSQL databases. Changes in the SQL database may not immediately reflect in Elasticsearch due to communication overhead. To address this, he introduced an experimental outbox polling coordination strategy, which uses additional SQL tables to maintain update order. While still in development, this feature promises to improve data consistency, a critical aspect for production environments.

Practical Applications and Benefits

Bartosz demonstrated practical applications of Hibernate Search through a bookstore example, where users could search for books by title, description, or reviews. His demo showed how to query Elasticsearch for terms like “Hibernate” or “programming,” retrieving relevant results ranked by relevance. Additionally, Hibernate Search supports advanced features like sorting by distance for geolocation-based queries and projections for retrieving partial documents, reducing reliance on the SQL database for certain operations. These capabilities make Hibernate Search a versatile tool for developers aiming to enhance search performance while maintaining their existing SQL infrastructure.

Links:

• BGŻ BNP Paribas company website

Bartek Gerlich, General Manager at 4SuITs Technology, delivered a candid talk at Devoxx Poland 2022 on the operational costs of running an IT company in Poland. Drawing from his experience building digital products for Caesars Entertainment and serving on the board of Plantwear, Bartek provided a detailed breakdown of costs at various company sizes, focusing on a B2B-focused limited liability company with a growth-oriented expense model.

Initial Setup Costs

Before establishing a company, key expenses include:

• Court Fees: Approximately 600 PLN for registration.
• Initial Capital: 5,000 PLN, which can be used for business expenses.
• Legal Fees: Around 1,000 PLN for a simple contract to ensure a smooth application process.
• Virtual Office: A few hundred PLN for a business address, avoiding personal address complications.

Costs with One Employee

Hiring the first employee introduces additional expenses:

• Legal Fees: Customized B2B contracts cost slightly more than boilerplate ones, but employment contracts significantly increase paperwork (e.g., work and safety regulations), tripling costs.
• Recruitment: For a junior developer, expect 10-20k PLN; mid-level 25k PLN; senior higher. These are ballpark figures for estimation.
• Accounting: Full accounting for a small company (10 documents/month) costs about 350 PLN.
• Equipment: Providing a laptop is advisable to protect intellectual property, costing around 3-5k PLN. Leasing reduces initial costs but increases long-term expenses.

Costs with Ten Employees

Scaling to a 10-person team, typically comprising five developers, two QAs, a project manager, a UX/UI designer, and a specialist (e.g., cloud engineer), incurs:

• Salaries: Developers and QAs average 21k PLN/month each; specialists around 30k PLN. A team manager or admin costs 7-8k PLN.
• Recruitment: External agencies charge 1-2 developer salaries per hire (e.g., 180k PLN for six hires). Recruitment process outsourcing (15k PLN/month) yields about two hires/month, while an internal recruiter (cheaper but slower) yields one hire/month.
• Office Options:
• Co-working: 2,500 PLN for occasional seats and conference rooms.
• Standalone Rental: 3,000 PLN, including utilities like coffee and electricity.
• Fully Managed Space: 4,000 PLN for four seats with shared amenities.
• Other Expenses:
• Legal Fees: 1,000 PLN for 8-10 hours/month of contract work.
• Accounting: 500 PLN for increased documentation.
• Equipment: 2,000 PLN for laptops, monitors, printers, etc., with 8-10% annual maintenance (e.g., 120k PLN total equipment yields 10-12k PLN/year maintenance).
• Utilities: Minimal, included in office costs.
• Total Monthly Cost: Approximately 250,000 PLN.

Costs with Fifty Employees

At 50 employees, the company resembles a scalable enterprise, with new roles like managers, enterprise sales reps, HR, and more senior admins:

• Salaries: Developers, QAs, PMs, QAs, UI/UX, and specialists continue, with managers and sales reps at ~30k PLN/month; senior admins at ~10K PLN; HR specialists at ~15k PLN. Ideally, 80% of staff generate revenue, with 20% in support roles, though middle management bloat can disrupt this.
• Recruitment: Costs scale with hires, with similar models (success-based, outsourcing, or internal).
• Office Costs:
• Standalone Rental: 30k PLN, requiring admin or security.
• Fully Managed Space: 50k PLN for 40 seats.
• A1/A1+ Commercial Space: 60k PLN (e.g., 15-20 EUR/sq.m in Warsaw, including shared spaces like toilets, corridors).
• Other Expenses:
• Legal Fees: 8-10k PLN/month for complex contracts.
• Accounting/Payroll: 8k PLN/month, higher for B2B contracts than employment contracts.
• Employee Benefits: 15k PLN/month for multisport, better coffee, or outings.
• Utilities: ~5k PLN/month.
• Travel: ~10k PLN/month for 10 travel days at 300 EUR/day.
• Total Monthly Cost: ~1.4 million PLN.

Scaling Beyond

Beyond 50 employees, costs scale linearly for office space, equipment, and recruitment, but non-linearly for salaries (due to increased management needs) and legal fees (due to disputes or complex contracts). Benefits and expenses also rise faster for larger team events or branding efforts.

Cost-Saving Strategies

• Small Teams (<10): Handle operations personally to save on admin/legal, use legal fees, and opt for fully remote to eliminate office costs, though admin logistics (e.g., contracts, equipment shipping) persist.
• Larger Teams:
• In-house Services: Internalize recruitment, admin, or legal services to reduce costs, though efficiency may suffer compared to third-party firms.
• Office Optimization: Use smaller, presentable spaces or hybrid models, but account for meeting/storage needs.
• Flat Hierarchy: Minimize middle management to maintain a lean structure.
• Junior Talent: Develop juniors in-house for cost savings, though it requires patience, with slower output initially.
• Software Tools: Use off-the-shelf solutions (e.g., Salesforce) with minimal customization to avoid expensive modifications.

Business Strategy Insights

Bartek addressed audience questions, noting:
– A healthy profit margin is ~20% to ensure cash flow and resilience against market shifts (e.g., recessions). Margins below ~7-10% are unsustainable.
– To avoid payment delays, secure credit lines or funding to maintain employee trust, as developers can easily find alternative employment elsewhere.
– Bootstrapping allows fast failure, validating ideas organically, but limits scale. Venture capital accelerates growth but requires strong pitching skills, often a challenge in Poland due to cultural gaps.
– Small businesses can succeed with modest profits (e.g., 600k PLN/year for a 10-person team at 10 people) without pursuing aggressive growth, unlike stock-market-driven firms needing constant expansion.

Conclusion

Running an IT business in Poland involves significant operational costs, dominated by salaries but with substantial non-profit-generating expenses (~20-40%). Strategic planning, cost optimization, and a clear growth vision are essential for profitability and sustainability. Bartek’s insights provide a practical guide for aspiring entrepreneurs navigating the financial realities of the IT sector.

Links:

• 4 Suits Technology company website
• Video

At Devoxx Poland 2022, Robert Wcisło and Łukasz Matug, data scientists at UBS, shared insights on ensuring the success of AI and NLP projects, drawing from their experience implementing AI solutions in a large investment bank. Their presentation highlighted critical success factors for deploying machine learning (ML) models into production, addressing common pitfalls and offering practical guidance across the project lifecycle.

Understanding the Challenges

The speakers noted that enthusiasm for AI often outpaces practical outcomes, with 2018 data indicating only 10% of ML projects reached production. While this figure may have improved, many projects still fail due to misaligned expectations or inadequate preparation. To counter this, they outlined a simplified three-phase process—Prepare, Build, and Maintain—integrating Software Development Lifecycle (SDLC) and MLOps principles, with a focus on delivering business value and user experience.

Prepare Phase: Setting the Foundation

Łukasz emphasized the importance of the Prepare phase, where clarity on business needs is critical. Many stakeholders, inspired by AI hype, expect miraculous solutions without defining specific outcomes. Key considerations include:

• Defining the Output: Understand the business problem and desired results, such as labeling outcomes (e.g., fraud detection). Reduce ambiguity by explicitly defining what the application should achieve.
• Evaluating ML Necessity: ML excels in areas like recommendation systems, language understanding, anomaly detection, and personalization, but it’s not a universal solution. For one-off problems, simpler analytics may suffice.
• Red Flags: ML models rarely achieve 100% accuracy, requiring more data and testing for higher precision, which increases costs. Highly regulated industries may demand transparency, posing challenges for complex models. Data availability is also critical—without sufficient data, ML is infeasible, though workarounds like transfer learning or purchasing data exist.
• Universal Performance Metric: Establish a metric aligned with business goals (e.g., click-through rate, precision/recall) to measure success, unify stakeholder expectations, and guide development priorities for cost efficiency.
• Tooling and Infrastructure: Align software and data science teams with shared tools (e.g., Git, data access, experiment logs). Ensure compliance with data restrictions (e.g., GDPR, cross-border rules) and secure access to production-like data and infrastructure (e.g., GPUs).
• Automation Levels: Decide the role of AI—ranging from no AI (human baseline) to full automation. Partial automation, where models handle clear cases and humans review uncertain ones, is often practical. Consider ethical principles like fairness, compliance, and no-harm to avoid bias or regulatory issues.
• Model Utilization: Plan how the model will be served—binary distribution, API service, embedded application, or self-service platform. Each approach impacts user experience, scalability, and maintenance.
• Scalability and Reuse: Design for scalability and consider reusing datasets or models to enhance future projects and reduce costs.

Build Phase: Crafting the Model

Robert focused on the Build phase, offering technical tips to streamline development:

• Data Management: Data evolves, requiring retraining to address drift. For NLP projects, cover diverse document templates, including slang or errors. Track data provenance and lineage to monitor sources and transformations, ensuring pipeline stability.
• Data Quality: Most ML projects involve smaller datasets (hundreds to thousands of points), where quality trumps quantity. Address imbalances by collaborating with clients for better data or using simpler models. Perform sanity checks to ensure representativeness, avoiding overly curated data that misaligns with production (e.g., professional photos vs. smartphone images).
• Metadata and Tagging: Use tags (e.g., source, date, document type) to simplify debugging and maintenance. For instance, identifying underperforming data (e.g., low-quality German PDFs) becomes easier with metadata.
• Labeling Strategy: Noisy or ambiguous labels (e.g., misinterpreting “bridges” as Jeff Bridges or drawings vs. physical bicycles) degrade model performance. Aim for human-level performance (HLP), either against ground truth (e.g., biopsy results) or inter-human agreement. A consistent labeling strategy, documented with clear examples, reduces ambiguity and improves data quality. Tools like AWS Mechanical Turk or in-house labeling platforms can streamline this process.
• Training Tips: Use transfer learning to leverage pre-trained models, reducing data needs. Active learning prioritizes labeling hard examples, while pseudo-labeling uses existing models to pre-annotate data, saving time if the model is reliable. Ensure determinism by fixing seeds for reproducibility during debugging. Start with lightweight models (e.g., BERT Tiny) to establish baselines before scaling to complex models.
• Baselines: Compare against prior models, heuristic-based systems, or simple proofs-of-concept to contextualize progress toward HLP. An 85% accuracy may be sufficient if it aligns with HLP, but 60% after extensive effort signals issues.

Maintain Phase: Sustaining Performance

Maintenance is critical as ML models differ from traditional software due to data drift and evolving inputs. Strategies include:

• Deployment Techniques: Use A/B testing to compare model versions, shadow mode to evaluate models in parallel with human processes, canary deployments to test on a small traffic subset, or blue-green deployments for seamless rollbacks.
• Monitoring: Beyond system metrics, monitor input (e.g., image brightness, speech volume, input length) and output (e.g., exact predictions, user behavior like query frequency). Detect data or concept drift to maintain relevance.
• Reuse: Reuse models, data, and experiences to reduce uncertainty, lower costs, and build organizational capabilities for future projects.

Key Takeaways

The speakers stressed reusing existing resources to demystify AI, reduce costs, and enhance efficiency. By addressing business needs, data quality, and operational challenges early, teams can increase the likelihood of delivering impactful AI-NLP solutions. They invited attendees to discuss further at the UBS stand, emphasizing practical application over theoretical magic.

Links:

• UBS company website
• Video of the talk

Leading in Tech – Michael Cullum

At GopherCon UK 2022, Michael Cullum, Head of Engineering at Bud, delivered an engaging talk on the multifaceted nature of leadership in the tech industry. With a wealth of experience in engineering leadership, Cullum explored what it means to be a leader, the diverse forms leadership takes, and how individuals can cultivate and identify effective leadership. His talk underscored that leadership is not confined to titles but is a universal opportunity to inspire and support others, making it a critical skill for all tech professionals.

Defining Leadership: Beyond Titles and Tasks

Cullum began by tackling the elusive definition of leadership, noting that even dictionaries and academic papers struggle to pin it down. He proposed that leadership is about promoting movement or change in others, not through coercion but by encouraging and supporting them. Unlike management, which often involves tasks like hiring or performance oversight, leadership focuses on the individuals being led, prioritizing their growth over the leader’s ego. Cullum emphasized that leadership is not about issuing orders but about fostering an environment where people are motivated to excel. This distinction is vital in tech, where roles like tech leads or managers can blur the line between task-oriented management and people-centric leadership.

Exploring Leadership Roles in Tech

Leadership in tech manifests in various forms, each with unique responsibilities. Cullum highlighted mentorship as a foundational leadership role, accessible to all regardless of seniority. Mentoring, whether formal or informal, involves sharing experiences to guide others, yet the industry often falls short in formalizing these relationships. Tech leads, another key role, translate business needs into technical direction but frequently focus on tasks like project management rather than inspiring their teams. Principal or staff engineers lead by example, serving as go-to experts who inspire through technical excellence. Public leaders, such as bloggers or conference speakers, drive change by sharing knowledge, while managers and senior leaders (e.g., CTOs) balance individual support with organizational goals. Cullum stressed that all these roles, when executed with a focus on others, embody leadership.

Traits of Effective Leaders

What makes a leader exceptional? Cullum outlined several critical traits. Listening—not just hearing but understanding—is paramount, as it fosters empathy and uncovers others’ needs. Leaders must communicate clearly, giving people time to digest complex ideas, and be mindful of power dynamics, speaking last in discussions to avoid stifling input. Generating energy and inspiring others, whether through actions or enthusiasm, is essential, as is maintaining a team-oriented mindset to avoid “us vs. them” divides. For tech leaders, staying technical—within reason—keeps them grounded, while managing team stress involves shielding members from undue pressure without hiding critical information. Cullum’s “poop analogy” illustrated this: great leaders act as umbrellas, filtering stress, not fans that scatter it chaotically.

Becoming and Finding Great Leaders

Cullum concluded with practical advice for aspiring leaders and those seeking them. Mentoring others, even informally, is the first step toward leadership, while seeking mentors outside one’s company provides unbiased guidance. Observing both good and bad leaders offers valuable lessons, and resources like books (e.g., The Manager’s Path by Camille Fournier) and communities like the Rands Leadership Slack enhance growth. When job hunting, Cullum urged asking about leadership style, vision, and team dynamics, as these outweigh transient tech stacks in importance. Great leaders respect, mentor, and prioritize your growth, fostering environments where you feel valued and inspired. By holding leaders to high standards and embracing leadership opportunities, everyone can contribute to a thriving tech ecosystem.

Links:

• Michael Cullum on LinkedIn
• Bud website
• Rands Leadership Slack

Hashtags: #Leadership #TechLeadership #Mentorship #GopherCon #MichaelCullum #Bud