Broadcom Test Data Manager (TDM) helps today’s agile teams overcome these challenges by allowing them to more easily manage their test data. And when integrated with TDM logs, Splunk enables real-time search, monitoring, and analysis for automated test data management at scale, giving teams broader data observability through Splunk’s powerful analytics and visualizations.

In this guide, you’ll learn how to load Broadcom TDM logs into the Splunk platform. By creating a Python script to process and ingest TDM logs for Splunk, you can make your test data logs easier to search, analyze for errors, and visualize for real-time monitoring and insights.

What Is Broadcom TDM?

Broadcom TDM is a solution that helps development teams and partners create and manage synthetic data for various testing purposes in their software development lifecycle (SDLC) processes. For example, teams can use TDM to create fit-for-purpose datasets for security and compliance validation, fill gaps in test data coverage, and create future scenarios and unexpected results to test boundary conditions, to name a few use cases.

Here are Broadcom TDM’s key features:

Synthetic Data Creation

TDM enables software teams to create synthetic test data from scratch using sophisticated coverage analysis, allowing for the smallest datasets required for comprehensive testing. With synthetic data, developers can test future scenarios and unexpected results—particularly values at the edge of input ranges, where errors are most likely to occur. 

This approach helps ensure that the software behaves as expected within the limits of allowable input, reducing the risk of costly delays and production errors.

Data Discovery and Profiling

TDM’s Discovery and Profiling feature provides data sampling and discovery capabilities that enable teams to profile data within a software project. These functionalities are crucial for identifying personally identifiable information (PII) across multiple data sources, ensuring that data privacy and compliance issues are surfaced and addressed as early as possible.

Data Masking and Virtual Test Data

Teams can use TDM to quickly provision virtual copies of test data for each tester on demand. This enables development and testing teams to deliver higher-quality software to market faster, at a lower cost, and with fewer bugs and deployment issues. 

TDM also provides a scalable PII masking feature that quickly and efficiently masks large data volumes, allowing teams to demonstrate continuous compliance with data privacy regulations without additional administrative or configuration overhead. 

Data Storage and Reuse

TDM allows software teams to store test data as centralized, reusable assets that can be cloned and provisioned in minutes to match specific tests and use cases. Data models and rules are also stored in a central repository, minimizing redundant efforts and maximizing the value of existing work.

Automated Test Data Management and Agile Software Development with Broadcom TDM

Comprehensive testing is essential to high-quality software, as it thoroughly evaluates all aspects of the application to uncover potential bugs, performance bottlenecks, and usability issues before they reach end users. Automated test data management is a critical component of comprehensive testing, enabling teams to scale their efforts efficiently and cost-effectively while achieving better results.

Agile software development relies on iterative, customer-centric, adaptive tactics to deliver high-quality software quickly and at a lower cost. TDM enhances data management in agile processes by reducing data volumes, test durations and costs, and overall development lifecycle time. 

Test data preparation efforts that extend beyond the typical two-week sprint cycle can disrupt a development team’s cadence—TDM helps avoid such delays by providing test data on demand, in seconds, for end-to-end application development and validation.

Why Integrate TDM with Splunk?

Integrating Broadcom TDM logs with Splunk offers significant advantages in test data management. Splunk enhances data visibility by providing powerful analytics and dashboards, giving your software development team deeper insights into your test data logs. This integration supports more efficient error monitoring and troubleshooting processes, improved compliance tracking with real-time insights, and timely access to accurate test data, helping your team align with the agile development schedule and cadence.

Broadcom TDM-Splunk Integration: Prerequisites

The following items are required for integrating your Broadcom TDM logs with Splunk:

• Splunk Enterprise
• Broadcom TDM log files
• Python IDE or code editor (e.g., PyCharm or Visual Studio Code)
• Some proficiency with Python

Please note that this guide specifically covers integrating Broadcom TDM log files with Splunk Enterprise (not Splunk Cloud) on a Windows-based system.

Loading TDM into Splunk: A Step-by-Step Guide

The following steps will show you how to set up Splunk to ingest TDM log files using a custom Python script.

Setting Up Splunk 

To configure your instance of Splunk Enterprise to ingest TDM log files, first create a new app. Navigate to the Apps section of Splunk Enterprise and click on the Create app button on the top-right side.

On the Add new page, provide a name for your new app (e.g., “Insert TDM Logs”).

Under Folder name, specify the app directory where your app will be located (e.g., “Insert_TDM_Logs”). Take note of the app’s directory path under Folder name—this is where you will place your custom Python script, in the bin subdirectory. The remaining fields are optional and can be left blank for now. 

Click the Save button to add your new app. Once the app has been saved, you will be redirected to the Apps page, where you can verify that your new app was created successfully.

Writing the Python Script to Process TDM Logs

The following Python script processes your TDM log files by first loading the log file from the folder specified in the dir variable (e.g., “C:\\data\\Tdm_logs\\logs\\”). The script then performs a pattern match for the word “ERROR” and outputs the results to a JSON-formatted file that Splunk can ingest and analyze.

import re
import os
import json
import sys

def filter_errors(log_line, filename):
    pattern = r"\[ERROR\]"
    debug_string_holder = {}
    match = re.search(pattern, log_line)

    if match:
        pattern = r"\s+"
        string_splits = re.split(pattern, log_line)
        debug_string_holder["Data"] = string_splits[0] + " " + string_splits[1]
        debug_string_holder["Type"] = string_splits[3]
        debug_string_holder["Message"] = " ".join(string_splits[4:])

    if len(debug_string_holder) != 0:
            json_object = json.dumps(debug_string_holder)
            print(json_object)

dir = "C:\\data\\Tdm_logs\\logs\\"
    for filename in os.listdir(dir):
        if filename.endswith(".log"):
            with open(dir+filename, 'r') as f:
                for line in f:
                    filter_errors(line, filename)

To modify the Python code for your system, copy and paste the Python code into your IDE or code editor, and replace the value of the dir variable with the path to your TDM log files. If you’re using Windows, remember to include an extra backslash to properly escape each backslash.

Save the Python script to the bin subdirectory of your app’s directory path (i.e., the Folder Name you specified earlier when creating the app) as “import_TDM_Debug_logs.py.” If using another file name, be sure to include the “.py” extension to identify it as a Python script.  

Automating Log Ingestion into Splunk

Once you have created your Python script, go back to the Apps page in Splunk Enterprise. Under Actions, click the Launch app link for the “Insert TDM Logs” app you created earlier.

In the top-level navigation, under Settings, click on Data inputs (under “DATA”). 

On the Data inputs page, click the Scripts link to add your Python script to your app.

You can configure your Python script’s settings on the Add Data page. Under Script Path, navigate to the bin directory where your Python script is saved.

For Script Name and Command, you can accept the default values provided by Splunk. You may also provide custom values for Interval Input and Interval to set the timing parameters that Splunk will use when executing your script.

Click Next to continue to the Input Settings page. For Source Type, select “Structured” and “json_no_timestamp.”  For App Context, make sure that the “Insert TDM Logs (Insert_TDM_Logs)” is selected. You can accept the default values for the other fields and click Next.

After clicking the Review and Submit buttons, your script input will be created. Click the Start Searching button to view your TDM error logs ingested into Splunk.

You can now view your TDM error logs in Splunk, filter by selected and interesting fields, and view Statistics and Visualizations by clicking on the respective tabs under the search pane.

Enhancing Test Data Management with Broadcom TDM

By following this guide, you can equip your development team with streamlined testing processes using Broadcom TDM logs and Splunk Enterprise, enabling real-time search, monitoring, and analysis for automated test data management at scale, along with data observability through analytics and visualizations.

For more information about integrating Test Data Manager by Broadcom or additional support, contact A&I Solutions with your questions today.

The post Integrating Broadcom Test Data Manager Logs with Splunk: A Step-by-Step Guide appeared first on A&I Solutions.

A Team of Certified Experts

As a longtime leader in the IT software and services industry, A&I is uniquely equipped to help you with whatever your technology challenges may be. Our entire tech team is certified in their respective fields, possessing both the knowledge and hands-on experience to help our customers achieve real, tangible results. In fact, two of our very own IT ninjas, Tim Higgins and Paul Stege, recently received Broadcom’s prestigious Knight Certification, joining a small and elite group of Technical Partner Professionals who have consistently demonstrated their product knowledge and are considered to be the best of the best. Read on below to find out more about the Broadcom Software Knight program and what sets A&I’s Knights apart from other IT professionals.

What is the Broadcom Software Knight Program?

The Broadcom Software Knights program is an invitation-only program designed to award and distinguish Broadcom partner consultants who have proven their technical expertise and who show a strong passion for Broadcom products. Broadcom describes Knights as “the most elite group of Partner Technical Professionals.” In fact, being named a Knight is the highest level of recognition that can be earned as a Broadcom partner. 

Knight Qualifications

Broadcom’s Software Knight title is bestowed on individuals who prove to be masterminds of Broadcom Software solutions. To qualify for knighthood, individuals must achieve the following:

• Proven sales and technical skills, including design, implementation, service, and/or support capabilities 
• Multiple certifications earned through a third-party exam verifying their understanding of key implementation and configuration concepts
• Show extensive sales expertise in Broadcom products and be skilled in performing proof-of-concepts
• Active participation in Broadcom Software trainings and events
• Frequent knowledge sharing with the Partner community at large

The Benefit of Working with a Broadcom Software Knight

Having an elite group of software gurus as your IT partner is a key step to effectively managing software solutions. When you work with a Knight-certified IT consultant, you’re guaranteed advanced professional support and guidance to help you architect the best technology solution for your business. Unlike other IT consultants, Knights know Broadcom software inside and out and can anticipate and plan for both present and future needs.

Meet Our Knights

A&I is extremely proud to have not one, but two, Knights on our team, both of whom hold multiple Knight certifications. Get to know our Knights below, and let us show you what working with a highly-skilled Knight can mean for your business.

Tim Higgins

Sr. Solutions Architect

Knighted in: Test Data Manager (TDM) and Application Performance Management (APM)  

Tim Higgins has been with A&I Solutions for over 11 years and has worked with software for over 30 years! Fueled by copious amounts of coffee, good music, and all things tech, Tim takes pride in utilizing his extensive IT background to support A&I’s diverse technical needs. While he has vast experience in many different software solutions, his specialty is DevOps and Monitoring products. From presales support to creating POCs and helping solve customers’ unique technical requests, Tim is always willing to lend a hand wherever he can. Working with software for 30 years has allowed Tim to understand Broadcom’s products from a business point of view as well as a developer’s.

Paul Stege

Principal Architect

Knighted in: DX NetOps and DX Unified Infrastructure Management (UIM)     

With over 20 years of experience developing and executing IT strategies, Paul Stege provides the hands-on support, technical knowledge, and forward-thinking vision to transform and simplify business processes while delivering bottom-line, measurable results. As a former CA Technologies IT executive, Paul knows technology inside and out and possesses both the technical expertise and business insight to drive better outcomes and help companies reach their strategic, long-term goals. At A&I Solutions specifically, he is the primary technical lead for our DX NetOps and DX UIM customers and has helped dozens of customers enhance their IT operations.

Achieve More Efficiency and Better Results 

Choosing to work with A&I and our insanely skilled Knights will ensure you get the most out of your technology investment. Tim, Paul, and the rest of our IT team are industry leaders who know how to leverage technology to drive success. Their long list of skills, wealth of experience, and understanding of Broadcom Software solutions are simply unmatched. 

Let us help you achieve more IT efficiency and stay ahead of the curve. Talk to one of our IT specialists to discover how we can help you achieve more.

The post Knighted and Ready To Serve appeared first on A&I Solutions.

TDM can help streamline your testing process by providing a centralized platform for managing your test data. If you’re unfamiliar with TDM, read on for a high-level overview of the product and its benefits.

What Is Test Data Manager (TDM)?

Test Data Manager is a software solution from Broadcom Software that helps users automate test data management and speeds up the delivery of test data. It provides a centralized platform for creating, storing, managing, and securing test data. TDM offers several features and benefits that can help streamline your testing process and improve your software quality.

What Can TDM Do?

Before we dive into the benefits of TDM, let’s take a look at what it can do for you. TDM offers features that can prove helpful in:

1. Identifying Sensitive Data

TDM has a TDM Discovery and Profiling feature that’s designed for identifying and classifying sensitive data. This feature can help you identify personal identifiable information (PII) and other types of sensitive data.

But why is this feature important? Well, data privacy is a huge concern for many organizations. And with the General Data Protection Regulation (GDPR) coming into effect, it’s more important than ever to be able to identify and protect sensitive data. Breaching compliance regulations can result in heavy fines, not to mention damaging your company’s reputation.

2. Data Masking

Another way that TDM can help you protect sensitive data is through data masking. Data masking is the process of hiding or changing certain, sensitive data elements. The purpose of data masking is to helps ensure that classified test data is not accidentally leaked.

With TDM, you can mask large quantities of data within minutes using automated data profiling and clone subsets of data, ensuring sensitive data remains secure.

3. Generating Test Data

In addition to identifying and masking the sensitive data in your system, TDM can help you generate synthetic test data that adheres to compliance regulations. This is important because it allows you to test your software without using actual production data. In other words, it allows testing teams to create realistic data tailored to their specific testing and development needs

This feature is also helpful in cases where you don’t have enough data to test with. For example, you might be developing a new feature that’s only used by a small subset of your users. In this case, it might not make sense to use actual production data to test with. TDM can help you generate the necessary test data from scratch.

4. Data Virtualization

Moreover, TDM also offers data virtualization. Data virtualization is the process of creating a virtual copy of your data. Test Data Manager allows for the rapid creation of virtual copies of large test data sets whenever needed. This ensures that each tester has their own personal copy of test data, instead of being forced to share a single copy of data which can slow down the entire development process. In short, the data virtualization feature of TDM provides testing teams with continuous access to consistent virtual data and environments so that they can stay on-pace with project timelines and delivery.

5. Data Storing and Reusing

Another way that TDM can help you streamline your testing process is by storing and reusing test data. This is beneficial because it allows teams to reuse any previously generated test data and work in parallel, eliminating the need for testers to create their own data. This is also helpful in cases where you need to run the same test multiple times.

The Benefits of Test Data Manager

Now that we’ve taken a look at what TDM can do, let’s take a look at some of the benefits it can bring to your organization. Here are some of them.

Time and Cost Savings

One of the biggest benefits of TDM is that it can help you save time and money. This is because it can help you automate your testing process. By automating your testing, you can free up your team’s time so that they can focus on other tasks.

In addition, automating your testing can also help you save money. This is because you won’t need to hire as many testers. You’ll also be able to avoid the cost of manually generating data.

Improved Quality and Efficiency

Another benefit of TDM is that it can help you improve the quality of your software. This is because TDM can help you catch errors early on in the development process by ensuring you are using the right test data which reduces defect creation. TDM can also help you improve the efficiency of your testing process through its automated features. This is because you won’t need to spend as much time setting up and running tests. In fact, some users of TDM report between 90 and 95 percent reduction in the amount of time they need for the provisioning of rich, top-notch test data.

Boost Your Software Testing with TDM

There’s no doubt that test data management is a critical part of the software development process. Test Data Manager by Broadcom Software can help you save time and money, improve the quality of your software, and improve the efficiency of your testing process. If you’re not already using TDM, now is the time to start!

Schedule a demo with A&I Solutions today to see TDM in action!

The post Why is Test Data Manager Essential in Software Testing? appeared first on A&I Solutions.

In the world of IT, this new teamwork spirit is best seen in SaaS companies. Traditionally, SaaS companies had software built by one team, and the developers’ team would pass it on to operations to run and operate it.

Tighter time-to-market windows have made this approach unsustainable because customers don’t want to spend long periods figuring out how to operate the software. Also, the market is flooded with intuitive software. A lack of collaboration between a company’s development and operations teams puts it at a disadvantage because they can’t work together to ensure the software performs optimally.

What DevOps Really Means

DevOps is a 21st-century term coined by merging development and operations. It represents a teamwork approach to the tasks performed by a company’s application development and IT operations teams.

An organization embracing DevOps also institutes a culture shift, such as building trust and cohesion between development and operations and intentionally aligning tech projects to business requirements.

Instead of building an app and figuring out the operational aspects later, you make operational requirements part of the engineering process. Early in development, you spot and build any additional elements needed to make the app resilient to meet customers’ needs.

Mistakes to Avoid in DevOps

DevOps provides a competitive advantage to any organization willing to embrace it, but there are several obstacles that can hinder businesses from reaping the full potential and benefits of DevOps. Watch the video below to find out what these pitfalls are.

Want to Learn More?

Check out the white paper below to get an in-depth overview on these common DevOps pitfalls and strategies on the best ways to avoid them.

The post 5 Common DevOps Pitfalls and Proven Ways to Avoid Them appeared first on A&I Solutions.

Why Test Data Manager?

• Delivers test data faster to help teams get the right data
• Enables rich synthetic data generation
• Easily identify PII data across multiple data sources

Learn more about Broadcom Test Data Manager by watching the video below. And when you’re ready, contact us to schedule a demo to see TDM in action.

The post Test Data Manager At a Glance appeared first on A&I Solutions.